2021-05-12T11:47:58.154 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-12T11:47:58.156 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_kchai@teuthology --description rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} --archive /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111458 --verbose --name kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi -- /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111458/orig.config.yaml 2021-05-12T11:47:58.187 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-12T11:47:58.231 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111458 branch: wip-kefu-testing-2021-05-12-1746 description: rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} email: null first_in_suite: false job_id: '6111458' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-05-12-1746 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2021-05-05-1449 sha1: 5cd0ad0001da189274219f5e829752717eb856d2 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-05-05-1449 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5cd0ad0001da189274219f5e829752717eb856d2 targets: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT+CYTwyX5zsnOgnT0gLNBfOMe4wHstV+X+Tul/G0o37+zsDZ7aS9m5rYO8Dsw46xOAFLn+Q6WKJJjXnOcUiJIDGHjKEvjfEXgksk6/cssQH2v5zTAjghil09mUvfkP5c+iXZbO8Vz/13m/7/mtb/bsrYHO+GUokE83JDh2+wb8lQgZ+BWOOaW2WZags/fsD/5iDTtke7Ydk4k8kcc3UgY6mAttjMXxiCTg+6FBrzmlzlmipY0p/0t78epS4I2K5YxfoEdd1yeHsjS1WvdYS/4o7oCsOI4j/c1HrJycejhWKmQ427xHlR0wDueeJlqAprQ6nPZZftzFz0K8jf9d9Y1wd4dFv21cTW9pDg5NY567/X/ubdtg/9vdRnPXWNZP2mWZXGMLWF2uAvhlLuRvSDQgIApSmYXaXVGgASX0Eb2PS+U63G3B9rTQIlRzfs02/QMX/DMckLCrr5cnkOu/5EZRXiBiZP00pSGi4m3JZ2h0ULPx6zk4fCUA9yoaIOFqUU= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: 19220a3bd6e252c6e8260827019668a766d85490 timestamp: 2021-05-12_11:35:51 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-12T11:47:58.231 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa; will attempt to use it 2021-05-12T11:47:58.232 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa/tasks 2021-05-12T11:47:58.233 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-12T11:47:58.234 INFO:teuthology.task.internal:Checking packages... 2021-05-12T11:47:58.257 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '36c2cf2fb22a9790b2c097e158985773b35f1740' 2021-05-12T11:47:58.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-12T11:47:58.258 INFO:teuthology.packaging:ref: None 2021-05-12T11:47:58.258 INFO:teuthology.packaging:tag: None 2021-05-12T11:47:58.259 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-05-12-1746 2021-05-12T11:47:58.259 INFO:teuthology.packaging:sha1: 36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:47:58.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-05-12-1746 2021-05-12T11:47:58.409 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-4087.g36c2cf2f 2021-05-12T11:47:58.410 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-12T11:47:58.421 INFO:teuthology.task.internal:no buildpackages task found 2021-05-12T11:47:58.421 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-12T11:47:58.467 INFO:teuthology.task.internal:Saving configuration 2021-05-12T11:47:58.479 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-12T11:47:58.504 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-12T11:47:58.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-12 11:42:25.223036', 'description': '/home/teuthworker/archive/kchai-2021-05-12_11:35:51-rados-wip-kefu-testing-2021-05-12-1746-distro-basic-smithi/6111458', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFoKmnZ9leenZBymTrmrLZcFLnm8vn4/znwHHj4FgxYUSwxi3WBUBkcC3gptUj8+aJ0y+9Ijf7vi4CZbLC8G28=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi090.front.sepia.ceph.com'} 2021-05-12T11:47:58.521 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-12T11:47:58.535 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-05-12T11:47:58.535 INFO:teuthology.run_tasks:Running task console_log... 2021-05-12T11:47:58.594 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efc5eaa7ea0>, signals=[15]) 2021-05-12T11:47:58.596 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-12T11:47:58.606 INFO:teuthology.task.internal:Opening connections... 2021-05-12T11:47:58.607 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:47:58.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:47:58.680 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-05-12T11:47:58.688 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2021-05-12T11:47:58.849 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2021-05-12T11:47:58.850 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2021-05-12T11:47:58.904 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Linux" 2021-05-12T11:47:58.905 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8" 2021-05-12T11:47:58.905 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2021-05-12T11:47:58.905 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2021-05-12T11:47:58.906 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2021-05-12T11:47:58.906 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2021-05-12T11:47:58.906 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-12T11:47:58.906 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2021-05-12T11:47:58.907 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-12T11:47:58.907 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2021-05-12T11:47:58.907 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-12T11:47:58.907 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-12T11:47:58.908 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-12T11:47:58.908 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-05-12T11:47:58.925 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-05-12T11:47:58.938 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-05-12T11:47:58.968 INFO:teuthology.task.internal:Checking for old test directory... 2021-05-12T11:47:58.969 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2021-05-12T11:47:58.983 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-05-12T11:47:59.003 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-05-12T11:47:59.004 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2021-05-12T11:47:59.050 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-05-12T11:47:59.077 INFO:teuthology.run_tasks:Running task kernel... 2021-05-12T11:47:59.100 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-05-12T11:47:59.101 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-05-12T11:47:59.101 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2021-05-12T11:47:59.116 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-05-12T11:47:59.116 INFO:teuthology.task.kernel:Running kernel on smithi090: 4.18.0-240.10.1.el8_3.x86_64 2021-05-12T11:47:59.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2021-05-12T11:48:03.679 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - AppStream 25 MB/s | 6.3 MB 00:00 2021-05-12T11:48:04.860 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - Base 13 MB/s | 2.3 MB 00:00 2021-05-12T11:48:05.322 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - Extras 111 kB/s | 9.6 kB 00:00 2021-05-12T11:48:07.220 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - AppStream 3.4 MB/s | 6.3 MB 00:01 2021-05-12T11:48:10.301 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - BaseOS 1.1 MB/s | 2.3 MB 00:02 2021-05-12T11:48:10.937 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - Extras 37 kB/s | 9.6 kB 00:00 2021-05-12T11:48:11.733 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 14 MB/s | 9.4 MB 00:00 2021-05-12T11:48:13.602 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-05-12T11:48:15.137 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-05-12T11:48:15.137 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-05-12T11:48:15.348 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:48:15.365 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:48:15.366 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:48:15.366 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:48:15.366 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:48:15.366 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 4.4 M 2021-05-12T11:48:15.367 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:48:15.367 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 30 M 2021-05-12T11:48:15.367 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 26 M 2021-05-12T11:48:15.367 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:15.367 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:48:15.368 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:48:15.368 INFO:teuthology.orchestra.run.smithi090.stdout:Install 3 Packages 2021-05-12T11:48:15.368 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:15.368 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 60 M 2021-05-12T11:48:15.369 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 83 M 2021-05-12T11:48:15.369 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:48:16.312 INFO:teuthology.orchestra.run.smithi090.stdout:(1/3): kernel-4.18.0-240.22.1.el8_3.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2021-05-12T11:48:17.366 INFO:teuthology.orchestra.run.smithi090.stdout:(2/3): kernel-modules-4.18.0-240.22.1.el8_3.x86 13 MB/s | 26 MB 00:01 2021-05-12T11:48:17.625 INFO:teuthology.orchestra.run.smithi090.stdout:(3/3): kernel-core-4.18.0-240.22.1.el8_3.x86_64 13 MB/s | 30 MB 00:02 2021-05-12T11:48:17.628 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:48:17.628 INFO:teuthology.orchestra.run.smithi090.stdout:Total 27 MB/s | 60 MB 00:02 2021-05-12T11:48:17.714 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:48:18.105 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:48:18.105 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:48:19.297 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:48:19.331 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:48:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:48:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T11:48:25.205 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T11:48:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T11:48:28.478 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T11:48:28.710 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T11:48:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T11:48:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T11:48:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T11:48:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T11:48:49.918 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T11:48:49.918 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:48:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:50.273 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:48:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:50.274 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:48:50.345 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel --last | head -n 1 2021-05-12T11:48:50.475 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Wed 12 May 2021 11:48:28 AM UTC 2021-05-12T11:48:50.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.477 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.477 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-05-12T11:48:50.478 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-05-12T11:48:50.478 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-05-12T11:48:50.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2021-05-12T11:48:50.479 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel --last | head -n 1 2021-05-12T11:48:50.587 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Wed 12 May 2021 11:48:28 AM UTC 2021-05-12T11:48:50.588 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2021-05-12T11:48:50.618 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2021-05-12T11:48:50.619 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2021-05-12T11:48:50.619 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.02 2021-05-12T11:48:50.619 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 90.el8 2021-05-12T11:48:50.620 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2021-05-12T11:48:50.620 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-05-12T11:48:50.620 INFO:teuthology.orchestra.run.smithi090.stdout:Group : System Environment/Base 2021-05-12T11:48:50.620 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 9197207 2021-05-12T11:48:50.621 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2021-05-12T11:48:50.621 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-05-12T11:48:50.621 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-05-12T11:48:50.621 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-05-12T11:48:50.622 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : kbuilder.bsys.centos.org 2021-05-12T11:48:50.622 INFO:teuthology.orchestra.run.smithi090.stdout:Relocations : (not relocatable) 2021-05-12T11:48:50.622 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : CentOS BuildSystem 2021-05-12T11:48:50.623 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2021-05-12T11:48:50.623 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2021-05-12T11:48:50.624 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2021-05-12T11:48:50.624 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2021-05-12T11:48:50.624 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:50.624 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-05-12T11:48:50.625 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2021-05-12T11:48:50.625 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2021-05-12T11:48:50.625 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2021-05-12T11:48:50.625 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:48:50.626 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2021-05-12T11:48:50.627 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-05-12T11:48:50.627 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:50.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-05-12T11:48:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2021-05-12T11:48:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:done 2021-05-12T11:48:55.244 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2021-05-12T11:48:55.279 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.EXjGwWrSz5 2021-05-12T11:48:55.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EXjGwWrSz5 2021-05-12T11:48:55.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.EXjGwWrSz5 2021-05-12T11:48:55.511 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.EXjGwWrSz5 is 6KB 2021-05-12T11:48:55.560 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.EXjGwWrSz5 2021-05-12T11:48:55.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2021-05-12T11:48:55.635 INFO:teuthology.orchestra.run.smithi090.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-05-12T11:48:55.636 INFO:teuthology.orchestra.run.smithi090.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-05-12T11:48:55.636 INFO:teuthology.orchestra.run.smithi090.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64.conf 2021-05-12T11:48:55.636 INFO:teuthology.orchestra.run.smithi090.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-05-12T11:48:55.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:48:55.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2021-05-12T11:49:25.820 INFO:teuthology.misc:Re-opening connections... 2021-05-12T11:49:25.821 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:25.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:25.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:44.267 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:49:44.268 DEBUG:teuthology.misc:waited 18.44706130027771 2021-05-12T11:49:45.269 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:45.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:45.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:48.331 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:49:48.331 DEBUG:teuthology.misc:waited 22.510361671447754 2021-05-12T11:49:49.332 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:49.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:49.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:51.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:49:51.407 DEBUG:teuthology.misc:waited 25.586344480514526 2021-05-12T11:49:52.408 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:52.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:52.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:54.475 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:49:54.475 DEBUG:teuthology.misc:waited 28.654682874679565 2021-05-12T11:49:55.477 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:55.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:55.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:49:57.547 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:49:57.547 DEBUG:teuthology.misc:waited 31.72638702392578 2021-05-12T11:49:58.548 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:49:58.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:49:58.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:00.623 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:00.623 DEBUG:teuthology.misc:waited 34.80237150192261 2021-05-12T11:50:01.624 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:01.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:01.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:03.691 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:03.691 DEBUG:teuthology.misc:waited 37.87064337730408 2021-05-12T11:50:04.693 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:04.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:04.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:06.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:06.767 DEBUG:teuthology.misc:waited 40.9464008808136 2021-05-12T11:50:07.769 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:07.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:07.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:09.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:09.835 DEBUG:teuthology.misc:waited 44.0144157409668 2021-05-12T11:50:10.837 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:10.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:10.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:12.911 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:12.911 DEBUG:teuthology.misc:waited 47.09046816825867 2021-05-12T11:50:13.913 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:13.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:13.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:50:15.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2021-05-12T11:50:15.979 DEBUG:teuthology.misc:waited 50.15841269493103 2021-05-12T11:50:16.981 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:50:16.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:50:16.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:51:17.041 DEBUG:teuthology.orchestra.remote:timed out 2021-05-12T11:51:17.043 DEBUG:teuthology.misc:waited 111.2223584651947 2021-05-12T11:51:18.045 INFO:teuthology.misc:trying to connect to ubuntu@smithi090.front.sepia.ceph.com 2021-05-12T11:51:18.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:51:18.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:51:18.354 DEBUG:teuthology.orchestra.run.smithi090:> true 2021-05-12T11:51:18.778 DEBUG:teuthology.misc:waited 112.95758152008057 2021-05-12T11:51:19.780 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-05-12T11:51:19.780 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.22.1.el8_3.x86_64"... 2021-05-12T11:51:19.781 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2021-05-12T11:51:19.798 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:51:19.798 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.22.1.el8_3.x86_64 vs 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T11:51:19.798 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-05-12T11:51:19.799 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-05-12T11:51:20.800 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-05-12T11:51:20.801 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-05-12T11:51:20.878 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2021-05-12T11:51:20.891 INFO:teuthology.run_tasks:Running task internal.base... 2021-05-12T11:51:20.900 INFO:teuthology.task.internal:Creating test directory... 2021-05-12T11:51:20.901 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-05-12T11:51:20.917 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-05-12T11:51:20.961 INFO:teuthology.run_tasks:Running task internal.archive... 2021-05-12T11:51:21.015 INFO:teuthology.task.internal:Creating archive directory... 2021-05-12T11:51:21.016 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-05-12T11:51:21.038 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-05-12T11:51:21.050 INFO:teuthology.task.internal:Enabling coredump saving... 2021-05-12T11:51:21.050 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-05-12T11:51:21.107 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:51:21.119 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:51:21.121 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-05-12T11:51:21.131 INFO:teuthology.task.internal:Configuring sudo... 2021-05-12T11:51:21.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-05-12T11:51:21.188 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-05-12T11:51:21.199 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-05-12T11:51:21.199 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-05-12T11:51:21.243 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T11:51:21.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T11:51:21.369 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T11:51:21.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T11:51:21.491 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:51:21.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-05-12T11:51:21.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2021-05-12T11:51:21.667 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-05-12T11:51:21.981 INFO:teuthology.run_tasks:Running task internal.timer... 2021-05-12T11:51:21.994 INFO:teuthology.task.internal:Starting timer... 2021-05-12T11:51:21.994 INFO:teuthology.run_tasks:Running task pcp... 2021-05-12T11:51:22.055 INFO:teuthology.run_tasks:Running task selinux... 2021-05-12T11:51:22.087 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-05-12T11:51:22.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2021-05-12T11:51:22.189 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: [ OK ] 2021-05-12T11:51:22.191 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-05-12T11:51:22.191 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2021-05-12T11:51:22.221 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2021-05-12T11:51:22.222 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2021-05-12T11:51:22.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-05-12T11:51:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T11:51:22.287 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-05-12T11:51:22.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2021-05-12T11:51:22.357 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-05-12T11:51:22.371 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-05-12T11:51:22.372 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-05-12T11:51:22.387 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-05-12T11:51:22.388 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-05-12T11:51:22.732 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-05-12T11:51:23.562 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-05-12T11:51:25.762 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-05-12T11:51:25.785 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:25 +0000 (0:00:00.040) 0:00:00.040 ********* 2021-05-12T11:51:27.282 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi090.front.sepia.ceph.com] 2021-05-12T11:51:27.334 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:27 +0000 (0:00:01.548) 0:00:01.589 ********* 2021-05-12T11:51:27.647 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-05-12T11:51:27.649 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T11:51:27.669 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:27 +0000 (0:00:00.335) 0:00:01.924 ********* 2021-05-12T11:51:28.253 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-05-12T11:51:28.255 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-05-12T11:51:28.274 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:28 +0000 (0:00:00.605) 0:00:02.530 ********* 2021-05-12T11:51:28.331 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-05-12T11:51:28.333 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:28.353 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:28 +0000 (0:00:00.078) 0:00:02.608 ********* 2021-05-12T11:51:28.667 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-05-12T11:51:28.669 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:51:28.689 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:28 +0000 (0:00:00.336) 0:00:02.944 ********* 2021-05-12T11:51:29.673 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-05-12T11:51:29.675 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820288.731145-20007-209177868410029/source", "state": "file", "uid": 0} 2021-05-12T11:51:29.696 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:29 +0000 (0:00:01.006) 0:00:03.951 ********* 2021-05-12T11:51:30.323 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-05-12T11:51:30.325 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-05-12T11:51:30.326 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-05-12T11:51:30.387 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-05-12T11:51:30.390 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-12T11:51:30.560 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:30 +0000 (0:00:00.864) 0:00:04.816 ********* 2021-05-12T11:51:32.185 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi090.front.sepia.ceph.com] 2021-05-12T11:51:32.236 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-05-12T11:51:32.238 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:32 +0000 (0:00:01.675) 0:00:06.491 ********* 2021-05-12T11:51:32.862 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-05-12T11:51:32.864 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.166120", "end": "2021-05-12 11:51:32.836002", "rc": 0, "start": "2021-05-12 11:51:32.669882", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-05-12T11:51:32.884 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:32 +0000 (0:00:00.647) 0:00:07.139 ********* 2021-05-12T11:51:33.075 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-05-12T11:51:33.077 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.002037", "end": "2021-05-12 11:51:33.050763", "failed_when_result": false, "rc": 0, "start": "2021-05-12 11:51:33.048726", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-05-12T11:51:33.099 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.215) 0:00:07.355 ********* 2021-05-12T11:51:33.168 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:33.190 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.090) 0:00:07.445 ********* 2021-05-12T11:51:33.257 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-05-12T11:51:33.259 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:33.279 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.089) 0:00:07.534 ********* 2021-05-12T11:51:33.383 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:33.404 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.125) 0:00:07.660 ********* 2021-05-12T11:51:33.482 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-05-12T11:51:33.484 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:33.508 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.104) 0:00:07.764 ********* 2021-05-12T11:51:33.577 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-05-12T11:51:33.578 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:51:33.599 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.090) 0:00:07.854 ********* 2021-05-12T11:51:33.831 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-05-12T11:51:33.833 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:51:33.855 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:51:33 +0000 (0:00:00.256) 0:00:08.110 ********* 2021-05-12T11:51:34.131 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-12T11:51:34.133 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-05-12T11:51:34.331 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-05-12T11:51:34.518 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-05-12T11:51:34.705 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-05-12T11:51:34.903 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-05-12T11:51:35.101 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-05-12T11:51:35.298 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-05-12T11:51:35.496 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-05-12T11:51:35.683 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-05-12T11:51:35.880 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-05-12T11:51:36.069 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-05-12T11:51:36.257 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-05-12T11:51:36.453 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-05-12T11:51:36.650 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-05-12T11:51:36.836 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-05-12T11:51:37.033 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-05-12T11:51:37.230 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-05-12T11:51:37.417 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-05-12T11:51:37.613 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-05-12T11:51:37.811 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-05-12T11:51:37.999 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-05-12T11:51:38.207 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-05-12T11:51:38.406 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-05-12T11:51:38.615 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-05-12T11:51:38.802 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-05-12T11:51:38.999 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-05-12T11:51:39.188 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-05-12T11:51:39.374 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-05-12T11:51:39.571 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-05-12T11:51:39.769 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-05-12T11:51:39.957 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-05-12T11:51:40.155 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-05-12T11:51:40.343 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-05-12T11:51:40.541 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-05-12T11:51:40.728 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-05-12T11:51:40.916 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-05-12T11:51:41.112 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-05-12T11:51:41.299 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-12T11:51:41.496 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-12T11:51:41.683 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-05-12T11:51:41.870 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-05-12T11:51:42.067 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-05-12T11:51:42.254 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-05-12T11:51:42.441 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-05-12T11:51:42.627 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-05-12T11:51:42.823 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-12T11:51:43.010 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-12T11:51:43.196 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-05-12T11:51:43.394 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-05-12T11:51:43.590 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-05-12T11:51:43.777 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-05-12T11:51:43.964 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-05-12T11:51:44.163 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-05-12T11:51:44.349 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-05-12T11:51:44.546 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-05-12T11:51:44.732 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-05-12T11:51:44.930 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-05-12T11:51:45.116 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-05-12T11:51:45.302 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-05-12T11:51:45.498 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-05-12T11:51:45.684 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-05-12T11:51:45.881 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-05-12T11:51:46.079 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-12T11:51:46.266 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-05-12T11:51:46.453 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-05-12T11:51:46.651 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-05-12T11:51:46.837 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-05-12T11:51:47.043 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-05-12T11:51:47.230 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-05-12T11:51:47.429 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-05-12T11:51:47.615 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-05-12T11:51:47.802 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-05-12T11:51:47.998 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-05-12T11:51:48.184 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-05-12T11:51:48.381 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-05-12T11:51:48.580 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-05-12T11:51:48.777 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-05-12T11:51:48.964 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-05-12T11:51:49.150 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-05-12T11:51:49.347 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-05-12T11:51:49.534 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-05-12T11:51:49.730 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-05-12T11:51:49.917 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-05-12T11:51:50.113 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-05-12T11:51:50.299 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-05-12T11:51:50.496 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-05-12T11:51:50.683 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-05-12T11:51:50.881 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-05-12T11:51:51.078 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-05-12T11:51:51.275 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-05-12T11:51:51.472 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-05-12T11:51:51.661 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-05-12T11:51:51.848 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-05-12T11:51:52.044 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-05-12T11:51:52.230 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-05-12T11:51:52.426 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-05-12T11:51:52.612 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-05-12T11:51:52.809 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-05-12T11:51:52.995 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-05-12T11:51:53.191 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-05-12T11:51:53.378 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-12T11:51:53.574 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-05-12T11:51:53.761 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-05-12T11:51:53.947 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-05-12T11:51:54.144 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-05-12T11:51:54.331 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-05-12T11:51:54.523 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-05-12T11:51:54.721 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-05-12T11:51:54.907 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-05-12T11:51:55.104 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-05-12T11:51:55.292 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-05-12T11:51:55.479 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-05-12T11:51:55.678 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-05-12T11:51:55.865 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-05-12T11:51:56.063 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-05-12T11:51:56.252 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-05-12T11:51:56.440 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-05-12T11:51:56.628 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-05-12T11:51:56.816 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-05-12T11:51:57.015 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-05-12T11:51:57.203 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-05-12T11:51:57.391 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-05-12T11:51:57.588 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-05-12T11:51:57.775 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-05-12T11:51:57.963 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-05-12T11:51:58.163 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-05-12T11:51:58.352 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-05-12T11:51:58.552 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-05-12T11:51:58.739 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-05-12T11:51:58.926 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-05-12T11:51:59.236 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-05-12T11:51:59.432 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-05-12T11:51:59.619 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-05-12T11:51:59.818 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-05-12T11:52:00.006 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-05-12T11:52:00.205 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-05-12T11:52:00.406 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-05-12T11:52:00.605 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-05-12T11:52:02.413 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-05-12T11:52:02.532 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:02 +0000 (0:00:28.677) 0:00:36.787 ********* 2021-05-12T11:52:02.713 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-05-12T11:52:02.734 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:02 +0000 (0:00:00.202) 0:00:36.990 ********* 2021-05-12T11:52:02.824 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-05-12T11:52:02.826 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-05-12T11:52:02.828 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-05-12T11:52:02.829 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-05-12T11:52:02.831 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-05-12T11:52:02.833 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5 2021-05-12T11:52:02.833 INFO:teuthology.task.ansible.out:mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-05-12T11:52:02.834 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d 2021-05-12T11:52:02.835 INFO:teuthology.task.ansible.out:7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumra 2021-05-12T11:52:02.837 INFO:teuthology.task.ansible.out:o", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b76 2021-05-12T11:52:02.838 INFO:teuthology.task.ansible.out:8c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70f 2021-05-12T11:52:02.840 INFO:teuthology.task.ansible.out:f00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52 2021-05-12T11:52:02.840 INFO:teuthology.task.ansible.out:345220522f8dd13c768bc5c1e41 2021-05-12T11:52:02.842 INFO:teuthology.task.ansible.out:e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5 2021-05-12T11:52:02.844 INFO:teuthology.task.ansible.out:dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b 2021-05-12T11:52:02.846 INFO:teuthology.task.ansible.out:1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}]}, "changed": false} 2021-05-12T11:52:02.867 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:02 +0000 (0:00:00.132) 0:00:37.123 ********* 2021-05-12T11:52:03.776 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-05-12T11:52:03.778 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"after": "2f6366fd31821994f3d217ad18c1675bf42a2115", "attempts": 1, "before": "2f6366fd31821994f3d217ad18c1675bf42a2115", "changed": false, "remote_url_changed": false} 2021-05-12T11:52:03.797 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:03 +0000 (0:00:00.930) 0:00:38.053 ********* 2021-05-12T11:52:04.112 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-12T11:52:04.114 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-05-12T11:52:04.320 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-05-12T11:52:04.527 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-05-12T11:52:04.838 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-05-12T11:52:04.840 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-05-12T11:52:05.035 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-05-12T11:52:05.242 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-05-12T11:52:05.244 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-05-12T11:52:05.438 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-05-12T11:52:05.645 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-05-12T11:52:05.852 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-05-12T11:52:05.854 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:06.060 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-05-12T11:52:06.061 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-05-12T11:52:06.258 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-05-12T11:52:06.259 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-05-12T11:52:06.269 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:06.280 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:06.479 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-05-12T11:52:06.678 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-05-12T11:52:06.680 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-05-12T11:52:06.886 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-05-12T11:52:07.083 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-05-12T11:52:07.085 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-05-12T11:52:07.289 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-05-12T11:52:07.497 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-05-12T11:52:07.714 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-05-12T11:52:07.931 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-05-12T11:52:07.933 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-05-12T11:52:08.138 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-05-12T11:52:08.140 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-05-12T11:52:08.346 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-05-12T11:52:08.553 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-05-12T11:52:08.761 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-05-12T11:52:08.763 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-05-12T11:52:08.968 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-05-12T11:52:09.175 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-05-12T11:52:09.372 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-05-12T11:52:09.569 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-05-12T11:52:09.776 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-05-12T11:52:09.973 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-05-12T11:52:09.975 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-05-12T11:52:10.180 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-05-12T11:52:10.378 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-05-12T11:52:10.574 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-05-12T11:52:10.781 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-05-12T11:52:10.977 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-05-12T11:52:11.174 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T11:52:11.380 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-05-12T11:52:11.578 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-05-12T11:52:11.784 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-05-12T11:52:11.786 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-05-12T11:52:11.992 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-05-12T11:52:11.994 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-05-12T11:52:12.188 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-05-12T11:52:12.384 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-05-12T11:52:12.602 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-05-12T11:52:12.798 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-05-12T11:52:12.800 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T11:52:13.006 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-05-12T11:52:13.008 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-05-12T11:52:13.010 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-05-12T11:52:13.212 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-05-12T11:52:13.419 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-05-12T11:52:13.626 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-05-12T11:52:13.823 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-05-12T11:52:14.030 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-05-12T11:52:14.228 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-05-12T11:52:14.230 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-05-12T11:52:14.438 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-05-12T11:52:14.638 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-05-12T11:52:14.847 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-05-12T11:52:14.849 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-05-12T11:52:15.046 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-05-12T11:52:15.255 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-05-12T11:52:15.464 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-05-12T11:52:15.465 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-05-12T11:52:15.661 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-05-12T11:52:15.860 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-05-12T11:52:16.060 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-05-12T11:52:16.268 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T11:52:16.466 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-05-12T11:52:16.675 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-05-12T11:52:16.874 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-05-12T11:52:17.071 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-05-12T11:52:17.279 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-05-12T11:52:17.478 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-05-12T11:52:17.675 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-05-12T11:52:17.893 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-05-12T11:52:18.090 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-05-12T11:52:18.298 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-05-12T11:52:18.497 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-05-12T11:52:18.705 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-05-12T11:52:18.902 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-05-12T11:52:19.101 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-05-12T11:52:19.308 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-05-12T11:52:19.504 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-05-12T11:52:19.712 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-05-12T11:52:19.713 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-05-12T11:52:19.908 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-05-12T11:52:20.116 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-05-12T11:52:20.313 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-05-12T11:52:20.521 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-05-12T11:52:20.719 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-05-12T11:52:20.721 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-05-12T11:52:20.936 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-05-12T11:52:20.938 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-05-12T11:52:21.133 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-05-12T11:52:21.341 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-05-12T11:52:21.538 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-05-12T11:52:21.747 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-05-12T11:52:21.997 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-05-12T11:52:22.291 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-05-12T11:52:22.488 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-05-12T11:52:22.695 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-05-12T11:52:22.697 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-05-12T11:52:22.902 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-05-12T11:52:23.109 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-05-12T11:52:23.316 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-05-12T11:52:23.523 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-05-12T11:52:23.720 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-05-12T11:52:23.916 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-05-12T11:52:24.123 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-05-12T11:52:24.125 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T11:52:24.319 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-05-12T11:52:24.536 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-05-12T11:52:24.732 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-05-12T11:52:24.940 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-05-12T11:52:25.146 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-05-12T11:52:25.354 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-05-12T11:52:25.551 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-05-12T11:52:25.757 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-05-12T11:52:25.954 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-05-12T11:52:26.161 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-05-12T11:52:26.369 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-05-12T11:52:26.565 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-05-12T11:52:26.771 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-05-12T11:52:26.988 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-05-12T11:52:27.196 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-05-12T11:52:27.402 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-05-12T11:52:27.599 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-05-12T11:52:27.806 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-05-12T11:52:28.022 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-05-12T11:52:28.024 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-05-12T11:52:28.220 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-05-12T11:52:28.416 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-05-12T11:52:28.623 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-05-12T11:52:28.819 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-05-12T11:52:29.025 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-05-12T11:52:29.232 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-05-12T11:52:29.428 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-05-12T11:52:29.634 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-05-12T11:52:29.841 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-05-12T11:52:30.047 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-05-12T11:52:30.243 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-05-12T11:52:30.450 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-05-12T11:52:30.646 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-05-12T11:52:30.853 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-05-12T11:52:31.049 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-05-12T11:52:31.266 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-05-12T11:52:31.473 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-05-12T11:52:31.475 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-05-12T11:52:31.680 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-05-12T11:52:31.886 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-05-12T11:52:32.063 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:32 +0000 (0:00:28.265) 0:01:06.318 ********* 2021-05-12T11:52:32.141 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-12T11:52:32.143 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.152 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.154 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.164 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.174 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.176 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.186 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.187 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.197 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.436 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-05-12T11:52:32.447 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.449 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.654 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-05-12T11:52:32.862 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-05-12T11:52:32.873 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.875 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.885 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.886 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.896 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.907 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.908 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.918 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.919 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.929 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.940 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.941 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.951 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.953 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.962 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.964 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.974 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.984 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.986 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.996 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:32.997 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.008 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.009 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.019 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.030 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.031 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.041 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.043 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.052 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.054 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.064 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.074 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.076 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.086 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.088 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.097 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.108 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.110 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.119 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.121 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.131 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.132 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.142 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.153 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.154 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.164 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.166 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.175 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.177 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.186 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.197 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.199 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.208 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.210 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.220 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.221 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.231 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.242 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.243 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.253 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.254 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.265 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.266 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.276 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.287 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.289 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.298 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.300 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.310 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.310 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.321 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.332 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.334 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.343 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.344 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.355 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.365 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.367 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.377 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.378 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.388 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.390 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.399 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.410 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.412 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.421 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.432 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.434 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.444 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.445 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.455 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.457 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.466 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.477 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.479 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.488 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.490 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.500 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.510 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.512 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.522 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.533 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.534 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.545 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.546 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.556 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.567 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.569 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.578 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.580 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.589 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.600 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.602 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.611 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.622 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.624 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.633 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.635 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.645 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.656 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.658 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.667 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.678 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.679 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.691 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.755 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:33 +0000 (0:00:01.692) 0:01:08.010 ********* 2021-05-12T11:52:33.824 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-05-12T11:52:33.826 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:33.845 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:33 +0000 (0:00:00.090) 0:01:08.100 ********* 2021-05-12T11:52:34.118 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-05-12T11:52:34.120 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-05-12T11:52:34.315 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-05-12T11:52:34.501 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-05-12T11:52:34.688 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-05-12T11:52:34.873 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-05-12T11:52:35.060 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-05-12T11:52:35.246 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-05-12T11:52:35.442 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-05-12T11:52:35.639 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-05-12T11:52:35.836 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-05-12T11:52:36.023 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-05-12T11:52:36.210 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-05-12T11:52:36.407 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-05-12T11:52:36.587 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-05-12T11:52:36.774 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-05-12T11:52:36.971 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-05-12T11:52:37.157 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-05-12T11:52:37.344 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-05-12T11:52:37.530 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-05-12T11:52:37.716 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-05-12T11:52:37.912 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-05-12T11:52:38.098 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-05-12T11:52:38.284 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-05-12T11:52:38.470 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-05-12T11:52:38.666 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-05-12T11:52:38.852 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-05-12T11:52:39.039 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-05-12T11:52:39.226 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-05-12T11:52:39.423 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-05-12T11:52:39.608 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-05-12T11:52:39.795 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-05-12T11:52:39.980 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-05-12T11:52:40.177 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-05-12T11:52:40.363 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-05-12T11:52:40.550 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-05-12T11:52:40.748 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-05-12T11:52:40.945 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-05-12T11:52:41.132 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-05-12T11:52:42.530 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-05-12T11:52:42.728 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-05-12T11:52:42.925 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-05-12T11:52:43.111 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-05-12T11:52:43.298 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-05-12T11:52:43.486 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-05-12T11:52:43.672 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-05-12T11:52:43.859 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-05-12T11:52:44.045 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-05-12T11:52:44.231 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-05-12T11:52:44.417 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-05-12T11:52:44.469 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:44 +0000 (0:00:10.623) 0:01:18.724 ********* 2021-05-12T11:52:45.001 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-05-12T11:52:45.003 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "checksum": "313dfb187fdfe6062591abb65b175f610cf74a8a", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "b87ac93b406a130b21ee0476c1efad05", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820364.5682373-25849-83989474456833/source", "state": "file", "uid": 0} 2021-05-12T11:52:45.023 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:45 +0000 (0:00:00.554) 0:01:19.279 ********* 2021-05-12T11:52:45.091 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-05-12T11:52:45.093 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => { "msg": "Host smithi090.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-05-12T11:52:45.114 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:45 +0000 (0:00:00.089) 0:01:19.369 ********* 2021-05-12T11:52:45.192 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-05-12T11:52:45.194 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-05-12T11:52:45.214 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:45 +0000 (0:00:00.100) 0:01:19.469 ********* 2021-05-12T11:52:45.281 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-05-12T11:52:45.283 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:45.304 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:45 +0000 (0:00:00.090) 0:01:19.560 ********* 2021-05-12T11:52:45.938 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-05-12T11:52:45.940 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.398162", "end": "2021-05-12 11:52:45.911547", "rc": 0, "start": "2021-05-12 11:52:45.513385", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-05-12T11:52:45.961 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:45 +0000 (0:00:00.656) 0:01:20.216 ********* 2021-05-12T11:52:46.028 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-05-12T11:52:46.030 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.050 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.089) 0:01:20.305 ********* 2021-05-12T11:52:46.117 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-05-12T11:52:46.119 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.139 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.088) 0:01:20.394 ********* 2021-05-12T11:52:46.206 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-05-12T11:52:46.208 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.228 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.089) 0:01:20.484 ********* 2021-05-12T11:52:46.296 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.319 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.090) 0:01:20.575 ********* 2021-05-12T11:52:46.387 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-05-12T11:52:46.389 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.415 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.096) 0:01:20.671 ********* 2021-05-12T11:52:46.483 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-05-12T11:52:46.485 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.505 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.089) 0:01:20.760 ********* 2021-05-12T11:52:46.591 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-05-12T11:52:46.592 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:52:46.613 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.107) 0:01:20.868 ********* 2021-05-12T11:52:46.691 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-05-12T11:52:46.692 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.713 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.099) 0:01:20.968 ********* 2021-05-12T11:52:46.780 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-05-12T11:52:46.781 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.802 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.089) 0:01:21.058 ********* 2021-05-12T11:52:46.870 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-05-12T11:52:46.871 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:46.897 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.094) 0:01:21.152 ********* 2021-05-12T11:52:46.965 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-05-12T11:52:46.967 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:52:46.988 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:46 +0000 (0:00:00.090) 0:01:21.243 ********* 2021-05-12T11:52:47.055 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-05-12T11:52:47.057 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.078 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.090) 0:01:21.333 ********* 2021-05-12T11:52:47.145 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-05-12T11:52:47.147 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:52:47.167 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.089) 0:01:21.422 ********* 2021-05-12T11:52:47.235 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.258 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.091) 0:01:21.514 ********* 2021-05-12T11:52:47.326 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-05-12T11:52:47.328 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.352 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.093) 0:01:21.607 ********* 2021-05-12T11:52:47.420 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-05-12T11:52:47.421 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.443 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.090) 0:01:21.697 ********* 2021-05-12T11:52:47.509 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-05-12T11:52:47.511 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.533 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.091) 0:01:21.788 ********* 2021-05-12T11:52:47.601 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-05-12T11:52:47.603 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:52:47.623 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.090) 0:01:21.878 ********* 2021-05-12T11:52:47.701 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-12T11:52:47.702 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.723 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.099) 0:01:21.978 ********* 2021-05-12T11:52:47.791 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.813 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.089) 0:01:22.068 ********* 2021-05-12T11:52:47.880 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-05-12T11:52:47.882 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:47.902 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:47 +0000 (0:00:00.089) 0:01:22.157 ********* 2021-05-12T11:52:47.980 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-05-12T11:52:47.981 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:48.005 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.102) 0:01:22.260 ********* 2021-05-12T11:52:48.072 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:48.093 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.088) 0:01:22.349 ********* 2021-05-12T11:52:48.171 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-05-12T11:52:48.173 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:48.193 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.099) 0:01:22.448 ********* 2021-05-12T11:52:48.260 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-05-12T11:52:48.262 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T11:52:48.282 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.089) 0:01:22.537 ********* 2021-05-12T11:52:48.350 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-05-12T11:52:48.352 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:48.373 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.090) 0:01:22.628 ********* 2021-05-12T11:52:48.454 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-05-12T11:52:48.456 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:48.477 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.103) 0:01:22.732 ********* 2021-05-12T11:52:48.740 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-05-12T11:52:48.742 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:52:48.763 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:48 +0000 (0:00:00.286) 0:01:23.018 ********* 2021-05-12T11:52:49.407 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-05-12T11:52:49.409 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-05-12T11:52:49.821 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-05-12T11:52:49.845 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:49 +0000 (0:00:01.081) 0:01:24.100 ********* 2021-05-12T11:52:50.335 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-05-12T11:52:50.337 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-05-12T11:52:50.751 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-05-12T11:52:50.775 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:50 +0000 (0:00:00.929) 0:01:25.030 ********* 2021-05-12T11:52:50.842 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-12T11:52:50.844 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:50.865 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:50 +0000 (0:00:00.090) 0:01:25.120 ********* 2021-05-12T11:52:50.933 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T11:52:50.935 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:50.955 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:50 +0000 (0:00:00.090) 0:01:25.211 ********* 2021-05-12T11:52:51.023 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-05-12T11:52:51.024 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.045 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.300 ********* 2021-05-12T11:52:51.112 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-05-12T11:52:51.114 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.134 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.390 ********* 2021-05-12T11:52:51.202 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-05-12T11:52:51.204 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.213 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.215 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.225 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.247 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.112) 0:01:25.502 ********* 2021-05-12T11:52:51.315 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T11:52:51.316 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.337 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.592 ********* 2021-05-12T11:52:51.404 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-05-12T11:52:51.406 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.427 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.682 ********* 2021-05-12T11:52:51.495 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.517 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.772 ********* 2021-05-12T11:52:51.584 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-05-12T11:52:51.586 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.606 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.861 ********* 2021-05-12T11:52:51.673 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-05-12T11:52:51.675 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:51.695 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:51 +0000 (0:00:00.089) 0:01:25.951 ********* 2021-05-12T11:52:52.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-05-12T11:52:52.522 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.639491", "end": "2021-05-12 11:52:52.500355", "rc": 0, "start": "2021-05-12 11:52:51.860864", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:52:52.543 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:52 +0000 (0:00:00.847) 0:01:26.798 ********* 2021-05-12T11:52:58.252 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-05-12T11:52:58.254 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:52:58.274 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:58 +0000 (0:00:05.730) 0:01:32.529 ********* 2021-05-12T11:52:58.342 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-05-12T11:52:58.344 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:58.364 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:58 +0000 (0:00:00.090) 0:01:32.620 ********* 2021-05-12T11:52:58.467 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-05-12T11:52:58.469 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:58.490 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:58 +0000 (0:00:00.125) 0:01:32.745 ********* 2021-05-12T11:52:58.590 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-05-12T11:52:58.592 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:58.613 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:58 +0000 (0:00:00.122) 0:01:32.868 ********* 2021-05-12T11:52:58.686 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:52:58.708 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:58 +0000 (0:00:00.095) 0:01:32.963 ********* 2021-05-12T11:52:59.183 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-05-12T11:52:59.185 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-05-12T11:52:59.205 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:52:59 +0000 (0:00:00.497) 0:01:33.461 ********* 2021-05-12T11:53:00.567 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-05-12T11:53:00.569 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:53:00.591 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:00 +0000 (0:00:01.385) 0:01:34.846 ********* 2021-05-12T11:53:01.108 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-12T11:53:01.110 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-05-12T11:53:01.526 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-05-12T11:53:01.939 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-05-12T11:53:01.963 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:01 +0000 (0:00:01.371) 0:01:36.218 ********* 2021-05-12T11:53:02.146 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-05-12T11:53:02.148 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:53:02.170 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:02 +0000 (0:00:00.207) 0:01:36.425 ********* 2021-05-12T11:53:02.657 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-12T11:53:02.659 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-05-12T11:53:03.059 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-05-12T11:53:03.463 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-05-12T11:53:03.486 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:03 +0000 (0:00:01.316) 0:01:37.742 ********* 2021-05-12T11:53:03.554 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-05-12T11:53:03.556 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:53:03.576 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:03 +0000 (0:00:00.090) 0:01:37.832 ********* 2021-05-12T11:53:03.655 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-05-12T11:53:03.657 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-05-12T11:53:03.678 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:03 +0000 (0:00:00.101) 0:01:37.933 ********* 2021-05-12T11:53:04.919 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-05-12T11:53:04.921 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-12T11:53:04.941 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:04 +0000 (0:00:01.263) 0:01:39.197 ********* 2021-05-12T11:53:05.019 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-05-12T11:53:05.021 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:53:05.031 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:53:05.032 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:53:05.042 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T11:53:05.065 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:05 +0000 (0:00:00.122) 0:01:39.320 ********* 2021-05-12T11:53:11.233 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-05-12T11:53:11.235 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64"]} 2021-05-12T11:53:11.256 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:11 +0000 (0:00:06.191) 0:01:45.511 ********* 2021-05-12T11:53:11.324 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-05-12T11:53:11.326 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:53:11.348 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:11 +0000 (0:00:00.092) 0:01:45.603 ********* 2021-05-12T11:53:11.837 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-05-12T11:53:11.839 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-05-12T11:53:11.860 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:11 +0000 (0:00:00.511) 0:01:46.115 ********* 2021-05-12T11:53:12.339 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-05-12T11:53:12.341 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-05-12T11:53:12.361 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:12 +0000 (0:00:00.501) 0:01:46.617 ********* 2021-05-12T11:53:12.438 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-05-12T11:53:12.440 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-05-12T11:53:12.461 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:12 +0000 (0:00:00.100) 0:01:46.717 ********* 2021-05-12T11:53:12.694 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-05-12T11:53:12.696 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:53:12.716 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:12 +0000 (0:00:00.254) 0:01:46.972 ********* 2021-05-12T11:53:13.032 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-05-12T11:53:13.034 INFO:teuthology.task.ansible.out:fatal: [smithi090.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-05-12T11:53:13.054 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:13 +0000 (0:00:00.337) 0:01:47.310 ********* 2021-05-12T11:53:13.123 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-05-12T11:53:13.124 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:53:13.145 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:13 +0000 (0:00:00.090) 0:01:47.400 ********* 2021-05-12T11:53:13.626 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-05-12T11:53:13.628 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-05-12T11:53:13.648 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:13 +0000 (0:00:00.503) 0:01:47.904 ********* 2021-05-12T11:53:14.357 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-05-12T11:53:14.359 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 11:53:09 UTC", "ActiveEnterTimestampMonotonic": "155811862", "ActiveExitTimestamp": "Wed 2021-05-12 11:53:09 UTC", "ActiveExitTimestampMonotonic": "155784054", "ActiveState": "active", "After": "var-run.mount -.mount time-sync.target systemd-tmpfiles-setup.service basic.target local-fs.target nss-lookup.target sysinit.target tmp.mount network-online.target system.slice systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 11:53:09 UTC", "AssertTimestampMonotonic": "155811091", "Before": "shutdown.target plymouth-quit.service xdm.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 11:53:09 UTC", "Co 2021-05-12T11:53:14.361 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "155811091", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "14938", "ExecMainStartTimestamp": "Wed 2021-05-12 11:53:09 UTC", "ExecMainStartTimestampMonotonic": "155811838", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Wed 2021-05-12 11:53:09 UTC", "InactiveEnterTimestampMonotonic": "155810415", "InactiveExitTimestamp": "Wed 2021-05-12 11:53:09 UTC", 2021-05-12T11:53:14.362 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "155811862", "InvocationID": "11917c670a09475da2ec5be15f7d0bdf", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "14938", "MemoryAccounting": "yes", "MemoryCurrent": "872448", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-05-12T11:53:14.364 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 11:53:09 UTC", "StateChangeTimestampMonotonic": "155811862", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 11:53:09 UTC", "WatchdogTimestampMonotonic": "155811861", "WatchdogUSec": "0"}} 2021-05-12T11:53:14.381 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:14 +0000 (0:00:00.732) 0:01:48.637 ********* 2021-05-12T11:53:14.605 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-05-12T11:53:14.607 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001701", "end": "2021-05-12 11:53:14.582415", "rc": 0, "start": "2021-05-12 11:53:14.580714", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-05-12T11:53:14.627 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:14 +0000 (0:00:00.245) 0:01:48.883 ********* 2021-05-12T11:53:15.987 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-05-12T11:53:15.989 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:53:16.008 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:16 +0000 (0:00:01.381) 0:01:50.264 ********* 2021-05-12T11:53:54.568 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-05-12T11:53:54.570 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch"]} 2021-05-12T11:53:54.591 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:54 +0000 (0:00:38.582) 0:02:28.846 ********* 2021-05-12T11:53:55.208 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-05-12T11:53:55.210 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-05-12T11:53:55.230 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:53:55 +0000 (0:00:00.638) 0:02:29.485 ********* 2021-05-12T11:54:11.671 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-05-12T11:54:11.673 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.215860", "end": "2021-05-12 11:54:11.648102", "failed_when_result": false, "rc": 0, "start": "2021-05-12 11:53:55.432242", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-05-12T11:54:11.695 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:11 +0000 (0:00:16.464) 0:02:45.950 ********* 2021-05-12T11:54:12.175 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-05-12T11:54:12.177 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-05-12T11:54:12.198 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:12 +0000 (0:00:00.503) 0:02:46.453 ********* 2021-05-12T11:54:12.430 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-05-12T11:54:12.432 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015688", "end": "2021-05-12 11:54:12.410282", "rc": 0, "start": "2021-05-12 11:54:12.394594", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:54:12.454 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:12 +0000 (0:00:00.256) 0:02:46.709 ********* 2021-05-12T11:54:12.696 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-05-12T11:54:12.698 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.026801", "end": "2021-05-12 11:54:12.679496", "rc": 0, "start": "2021-05-12 11:54:12.652695", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:54:12.718 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:12 +0000 (0:00:00.264) 0:02:46.973 ********* 2021-05-12T11:54:29.362 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-05-12T11:54:29.363 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.422294", "end": "2021-05-12 11:54:29.336240", "rc": 0, "start": "2021-05-12 11:54:12.913946", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:54:29.383 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:29 +0000 (0:00:16.665) 0:03:03.639 ********* 2021-05-12T11:54:29.617 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-05-12T11:54:29.619 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-05-12T11:54:29.640 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:29 +0000 (0:00:00.256) 0:03:03.895 ********* 2021-05-12T11:54:29.708 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-05-12T11:54:29.710 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-05-12T11:54:29.731 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:29 +0000 (0:00:00.090) 0:03:03.986 ********* 2021-05-12T11:54:29.809 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-05-12T11:54:29.810 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T11:54:29.832 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:29 +0000 (0:00:00.101) 0:03:04.088 ********* 2021-05-12T11:54:29.911 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-05-12T11:54:29.913 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-05-12T11:54:29.936 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:29 +0000 (0:00:00.103) 0:03:04.191 ********* 2021-05-12T11:54:30.004 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-05-12T11:54:30.006 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T11:54:30.039 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.102) 0:03:04.294 ********* 2021-05-12T11:54:30.220 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-05-12T11:54:30.221 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T11:54:30.242 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.203) 0:03:04.498 ********* 2021-05-12T11:54:30.476 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-05-12T11:54:30.478 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-05-12T11:54:30.498 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.255) 0:03:04.753 ********* 2021-05-12T11:54:30.566 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-05-12T11:54:30.568 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:30.587 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.088) 0:03:04.842 ********* 2021-05-12T11:54:30.654 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-05-12T11:54:30.655 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:30.676 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.089) 0:03:04.931 ********* 2021-05-12T11:54:30.929 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-05-12T11:54:30.931 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-05-12T11:54:30.950 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:30 +0000 (0:00:00.273) 0:03:05.205 ********* 2021-05-12T11:54:31.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-05-12T11:54:31.506 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-05-12T11:54:31.526 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:31 +0000 (0:00:00.576) 0:03:05.782 ********* 2021-05-12T11:54:31.604 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-05-12T11:54:31.606 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi090"}, "changed": false} 2021-05-12T11:54:31.629 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:31 +0000 (0:00:00.102) 0:03:05.884 ********* 2021-05-12T11:54:32.666 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-05-12T11:54:32.668 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi090.front.sepia.ceph.com", "ansible_hostname": "smithi090", "ansible_nodename": "smithi090"}, "changed": true, "name": "smithi090"} 2021-05-12T11:54:32.688 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:32 +0000 (0:00:01.059) 0:03:06.944 ********* 2021-05-12T11:54:33.157 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-05-12T11:54:33.159 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-05-12T11:54:33.179 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:33 +0000 (0:00:00.491) 0:03:07.435 ********* 2021-05-12T11:54:33.663 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-05-12T11:54:33.664 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-05-12T11:54:33.685 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:33 +0000 (0:00:00.505) 0:03:07.940 ********* 2021-05-12T11:54:34.289 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-05-12T11:54:34.290 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-05-12T11:54:34.311 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:34 +0000 (0:00:00.625) 0:03:08.566 ********* 2021-05-12T11:54:34.800 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-05-12T11:54:34.802 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-05-12T11:54:34.824 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:34 +0000 (0:00:00.513) 0:03:09.079 ********* 2021-05-12T11:54:35.046 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-12T11:54:35.048 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T11:54:35.068 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:35 +0000 (0:00:00.244) 0:03:09.323 ********* 2021-05-12T11:54:35.301 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-12T11:54:35.302 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-05-12T11:54:35.324 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:35 +0000 (0:00:00.255) 0:03:09.579 ********* 2021-05-12T11:54:35.577 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-12T11:54:35.578 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-05-12T11:54:35.599 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:35 +0000 (0:00:00.275) 0:03:09.854 ********* 2021-05-12T11:54:36.088 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T11:54:36.090 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-05-12T11:54:36.110 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:36 +0000 (0:00:00.511) 0:03:10.365 ********* 2021-05-12T11:54:36.599 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T11:54:36.601 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-05-12T11:54:36.622 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:36 +0000 (0:00:00.511) 0:03:10.877 ********* 2021-05-12T11:54:39.234 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-12T11:54:39.236 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.388313", "end": "2021-05-12 11:54:39.209344", "rc": 0, "start": "2021-05-12 11:54:36.821031", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:54:39.257 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:39 +0000 (0:00:02.635) 0:03:13.512 ********* 2021-05-12T11:54:39.761 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-05-12T11:54:39.763 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-05-12T11:54:40.166 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-05-12T11:54:40.579 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-05-12T11:54:40.602 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:40 +0000 (0:00:01.345) 0:03:14.857 ********* 2021-05-12T11:54:41.095 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-05-12T11:54:41.097 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-05-12T11:54:41.119 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:41 +0000 (0:00:00.516) 0:03:15.374 ********* 2021-05-12T11:54:41.602 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-05-12T11:54:41.603 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-05-12T11:54:42.016 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-05-12T11:54:42.428 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-05-12T11:54:42.452 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:01.333) 0:03:16.708 ********* 2021-05-12T11:54:42.509 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-05-12T11:54:42.531 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.078) 0:03:16.786 ********* 2021-05-12T11:54:42.599 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-05-12T11:54:42.601 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:42.632 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.100) 0:03:16.887 ********* 2021-05-12T11:54:42.713 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-05-12T11:54:42.715 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-12T11:54:42.736 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.103) 0:03:16.991 ********* 2021-05-12T11:54:42.803 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-05-12T11:54:42.805 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:42.824 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.088) 0:03:17.080 ********* 2021-05-12T11:54:42.891 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:54:42.893 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-05-12T11:54:42.913 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.089) 0:03:17.169 ********* 2021-05-12T11:54:42.980 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-05-12T11:54:42.982 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:43.002 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:42 +0000 (0:00:00.088) 0:03:17.257 ********* 2021-05-12T11:54:43.069 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:43.090 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:43 +0000 (0:00:00.088) 0:03:17.345 ********* 2021-05-12T11:54:43.331 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-05-12T11:54:43.333 INFO:teuthology.task.ansible.out:fatal: [smithi090.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007005", "end": "2021-05-12 11:54:43.311364", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 11:54:43.304359", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-05-12T11:54:43.354 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:43 +0000 (0:00:00.263) 0:03:17.609 ********* 2021-05-12T11:54:43.420 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-05-12T11:54:43.422 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:43.444 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:43 +0000 (0:00:00.090) 0:03:17.699 ********* 2021-05-12T11:54:46.336 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-12T11:54:46.337 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-05-12T11:54:46.358 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:46 +0000 (0:00:02.914) 0:03:20.613 ********* 2021-05-12T11:54:49.013 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-12T11:54:49.014 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-05-12T11:54:49.034 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:49 +0000 (0:00:02.676) 0:03:23.289 ********* 2021-05-12T11:54:56.096 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T11:54:56.098 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64"]} 2021-05-12T11:54:56.119 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:56 +0000 (0:00:07.084) 0:03:30.374 ********* 2021-05-12T11:54:56.191 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-05-12T11:54:56.193 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:54:56.214 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:56 +0000 (0:00:00.094) 0:03:30.469 ********* 2021-05-12T11:54:57.707 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-12T11:54:57.709 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:54:57.729 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:57 +0000 (0:00:01.515) 0:03:31.984 ********* 2021-05-12T11:54:59.107 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:54:59.109 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:54:59.128 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:59 +0000 (0:00:01.399) 0:03:33.383 ********* 2021-05-12T11:54:59.453 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-05-12T11:54:59.454 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service basic.target polkit.service dbus.socket system.slice sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service nftables.service ipset.service iptables.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-05-12T11:54:59.456 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-05-12T11:54:59.457 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-05-12T11:54:59.459 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-05-12T11:54:59.477 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:59 +0000 (0:00:00.348) 0:03:33.732 ********* 2021-05-12T11:54:59.796 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-05-12T11:54:59.798 INFO:teuthology.task.ansible.out:fatal: [smithi090.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-05-12T11:54:59.818 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:54:59 +0000 (0:00:00.341) 0:03:34.073 ********* 2021-05-12T11:55:00.515 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-05-12T11:55:00.517 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-05-12T11:55:00.546 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.728) 0:03:34.801 ********* 2021-05-12T11:55:00.605 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-05-12T11:55:00.607 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:00.627 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.080) 0:03:34.882 ********* 2021-05-12T11:55:00.694 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-05-12T11:55:00.696 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:00.716 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.089) 0:03:34.971 ********* 2021-05-12T11:55:00.783 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-05-12T11:55:00.785 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:00.806 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.090) 0:03:35.061 ********* 2021-05-12T11:55:00.873 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-05-12T11:55:00.875 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:00.896 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.089) 0:03:35.151 ********* 2021-05-12T11:55:00.963 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-05-12T11:55:00.965 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:00.986 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:00 +0000 (0:00:00.090) 0:03:35.241 ********* 2021-05-12T11:55:01.054 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-05-12T11:55:01.056 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.079 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.092) 0:03:35.334 ********* 2021-05-12T11:55:01.137 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-05-12T11:55:01.158 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.079) 0:03:35.413 ********* 2021-05-12T11:55:01.215 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-05-12T11:55:01.216 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.236 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.078) 0:03:35.492 ********* 2021-05-12T11:55:01.466 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.089) 0:03:35.581 ********* TASK [testnode : Install apt keys] ********************************************* skipping: [smithi090.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.100) 0:03:35.681 ********* 2021-05-12T11:55:01.483 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-05-12T11:55:01.484 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.505 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.078) 0:03:35.760 ********* 2021-05-12T11:55:01.572 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-05-12T11:55:01.594 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.088) 0:03:35.849 ********* 2021-05-12T11:55:01.662 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-05-12T11:55:01.663 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.683 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.089) 0:03:35.938 ********* 2021-05-12T11:55:01.750 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-05-12T11:55:01.752 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.772 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.089) 0:03:36.027 ********* 2021-05-12T11:55:01.839 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-05-12T11:55:01.841 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.860 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.088) 0:03:36.116 ********* 2021-05-12T11:55:01.918 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:55:01.920 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:01.938 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:01 +0000 (0:00:00.077) 0:03:36.194 ********* 2021-05-12T11:55:02.006 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-05-12T11:55:02.007 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.028 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.089) 0:03:36.283 ********* 2021-05-12T11:55:02.095 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T11:55:02.097 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.120 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.092) 0:03:36.375 ********* 2021-05-12T11:55:02.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-05-12T11:55:02.179 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.199 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.079) 0:03:36.454 ********* 2021-05-12T11:55:02.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-05-12T11:55:02.268 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.288 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.088) 0:03:36.543 ********* 2021-05-12T11:55:02.345 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-05-12T11:55:02.347 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.366 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.078) 0:03:36.622 ********* 2021-05-12T11:55:02.423 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-05-12T11:55:02.424 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.445 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.078) 0:03:36.700 ********* 2021-05-12T11:55:02.501 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-05-12T11:55:02.503 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.522 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.076) 0:03:36.777 ********* 2021-05-12T11:55:02.588 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-05-12T11:55:02.590 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.610 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.088) 0:03:36.865 ********* 2021-05-12T11:55:02.677 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-05-12T11:55:02.679 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.699 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.088) 0:03:36.954 ********* 2021-05-12T11:55:02.756 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-05-12T11:55:02.757 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.777 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.078) 0:03:37.033 ********* 2021-05-12T11:55:02.844 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.866 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.088) 0:03:37.121 ********* 2021-05-12T11:55:02.933 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-12T11:55:02.935 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:02.954 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:02 +0000 (0:00:00.087) 0:03:37.209 ********* 2021-05-12T11:55:03.020 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-12T11:55:03.022 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.042 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.087) 0:03:37.297 ********* 2021-05-12T11:55:03.109 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-12T11:55:03.111 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.133 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.091) 0:03:37.388 ********* 2021-05-12T11:55:03.190 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T11:55:03.191 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.211 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.078) 0:03:37.467 ********* 2021-05-12T11:55:03.268 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T11:55:03.269 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.289 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.077) 0:03:37.545 ********* 2021-05-12T11:55:03.357 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.378 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.089) 0:03:37.634 ********* 2021-05-12T11:55:03.435 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.457 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.078) 0:03:37.712 ********* 2021-05-12T11:55:03.524 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-12T11:55:03.526 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.546 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.089) 0:03:37.801 ********* 2021-05-12T11:55:03.616 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:03.617 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.638 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.091) 0:03:37.893 ********* 2021-05-12T11:55:03.706 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-12T11:55:03.708 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.728 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.090) 0:03:37.983 ********* 2021-05-12T11:55:03.796 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.817 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.089) 0:03:38.073 ********* 2021-05-12T11:55:03.884 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T11:55:03.886 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.906 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.088) 0:03:38.161 ********* 2021-05-12T11:55:03.974 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T11:55:03.976 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:03.995 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:03 +0000 (0:00:00.089) 0:03:38.251 ********* 2021-05-12T11:55:04.239 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T11:55:04.240 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-05-12T11:55:04.260 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:04 +0000 (0:00:00.264) 0:03:38.515 ********* 2021-05-12T11:55:04.494 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:55:04.496 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:55:04.516 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:04 +0000 (0:00:00.256) 0:03:38.772 ********* 2021-05-12T11:55:04.750 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T11:55:04.752 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-05-12T11:55:04.772 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:04 +0000 (0:00:00.255) 0:03:39.027 ********* 2021-05-12T11:55:04.839 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-12T11:55:04.841 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:04.861 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:04 +0000 (0:00:00.089) 0:03:39.116 ********* 2021-05-12T11:55:04.929 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-12T11:55:04.931 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:04.951 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:04 +0000 (0:00:00.090) 0:03:39.207 ********* 2021-05-12T11:55:05.040 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T11:55:05.041 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.062 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.110) 0:03:39.317 ********* 2021-05-12T11:55:05.129 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T11:55:05.131 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.151 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.088) 0:03:39.406 ********* 2021-05-12T11:55:05.217 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:55:05.219 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.239 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.088) 0:03:39.495 ********* 2021-05-12T11:55:05.306 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T11:55:05.308 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.328 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.088) 0:03:39.584 ********* 2021-05-12T11:55:05.395 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T11:55:05.397 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.417 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.088) 0:03:39.672 ********* 2021-05-12T11:55:05.484 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T11:55:05.485 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.505 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.088) 0:03:39.760 ********* 2021-05-12T11:55:05.572 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.593 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.087) 0:03:39.848 ********* 2021-05-12T11:55:05.660 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-05-12T11:55:05.663 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.685 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.091) 0:03:39.940 ********* 2021-05-12T11:55:05.752 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-05-12T11:55:05.754 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.774 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.089) 0:03:40.029 ********* 2021-05-12T11:55:05.841 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-12T11:55:05.843 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.864 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.089) 0:03:40.119 ********* 2021-05-12T11:55:05.921 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-12T11:55:05.922 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:05.942 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:05 +0000 (0:00:00.078) 0:03:40.197 ********* 2021-05-12T11:55:05.999 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-05-12T11:55:06.001 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.021 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.079) 0:03:40.277 ********* 2021-05-12T11:55:06.089 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-05-12T11:55:06.090 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.118 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.096) 0:03:40.374 ********* 2021-05-12T11:55:06.188 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-05-12T11:55:06.189 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.208 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.090) 0:03:40.464 ********* 2021-05-12T11:55:06.265 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-05-12T11:55:06.266 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.286 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.077) 0:03:40.541 ********* 2021-05-12T11:55:06.353 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-05-12T11:55:06.355 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.375 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.088) 0:03:40.630 ********* 2021-05-12T11:55:06.452 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-05-12T11:55:06.454 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.474 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.099) 0:03:40.729 ********* 2021-05-12T11:55:06.542 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-05-12T11:55:06.543 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.564 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.089) 0:03:40.819 ********* 2021-05-12T11:55:06.630 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-05-12T11:55:06.632 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.652 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.088) 0:03:40.908 ********* 2021-05-12T11:55:06.719 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-05-12T11:55:06.721 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.743 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.090) 0:03:40.999 ********* 2021-05-12T11:55:06.816 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:06.818 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-05-12T11:55:06.840 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.096) 0:03:41.095 ********* 2021-05-12T11:55:06.917 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-05-12T11:55:06.919 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.928 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-05-12T11:55:06.930 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.939 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.941 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.951 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:06.977 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:06 +0000 (0:00:00.137) 0:03:41.232 ********* 2021-05-12T11:55:07.199 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-05-12T11:55:07.201 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003333", "end": "2021-05-12 11:55:07.180305", "rc": 0, "start": "2021-05-12 11:55:07.176972", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock"]} 2021-05-12T11:55:07.224 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:07 +0000 (0:00:00.246) 0:03:41.479 ********* 2021-05-12T11:55:07.290 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-05-12T11:55:07.292 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:07.315 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:07 +0000 (0:00:00.090) 0:03:41.570 ********* 2021-05-12T11:55:07.381 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-05-12T11:55:07.383 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:07.404 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:07 +0000 (0:00:00.088) 0:03:41.659 ********* 2021-05-12T11:55:07.471 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-05-12T11:55:07.473 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:07.493 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:07 +0000 (0:00:00.089) 0:03:41.748 ********* 2021-05-12T11:55:07.560 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-05-12T11:55:07.562 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:07.582 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:07 +0000 (0:00:00.089) 0:03:41.837 ********* 2021-05-12T11:55:08.948 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-05-12T11:55:08.950 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:55:08.970 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:08 +0000 (0:00:01.388) 0:03:43.225 ********* 2021-05-12T11:55:09.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-12T11:55:09.049 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278354944, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118739, 'block_used': 12976369, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118739, "block_size": 4096, "block_total": 240095108, "block_used": 12976369, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278354944, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-05-12T11:55:09.073 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:09 +0000 (0:00:00.102) 0:03:43.328 ********* 2021-05-12T11:55:09.296 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-05-12T11:55:09.298 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005032", "end": "2021-05-12 11:55:09.271808", "rc": 0, "start": "2021-05-12 11:55:09.266776", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:09.317 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:09 +0000 (0:00:00.244) 0:03:43.573 ********* 2021-05-12T11:55:09.395 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-05-12T11:55:09.397 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278354944, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118739, 'block_used': 12976369, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118739, "block_size": 4096, "block_total": 240095108, "block_used": 12976369, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278354944, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:09.418 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:09 +0000 (0:00:00.100) 0:03:43.673 ********* 2021-05-12T11:55:09.475 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-05-12T11:55:09.496 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:09 +0000 (0:00:00.078) 0:03:43.751 ********* 2021-05-12T11:55:11.348 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-05-12T11:55:11.349 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBDFzSGHcoMi7gSeZeFecoGNE3dTOTWtCJ'], 'uuids': ['0c2ecc44-086e-4bf5-a151-ce146ec2d80b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.609548", "end": "2021-05-12 11:55:11.314491", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBDFzSGHcoMi7gSeZeFecoGNE3dTOTWtCJ"], "labels": [], "masters": [], "uuids": ["0c2ecc44-086e-4bf5-a151-ce146ec2d80b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:09.704943", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:12.598 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-2swM1z-zau7-hgjc-OoT7-C5hZ-0eOT-JgX5lv', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UK400BGN', 'nvme-nvme.8086-50484654363230343030554b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.075300", "end": "2021-05-12 11:55:12.568612", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2swM1z-zau7-hgjc-OoT7-C5hZ-0eOT-JgX5lv", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UK400BGN", "nvme-nvme.8086-50484654363230343030554b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:11.493312", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-05-12T11:55:12.600 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:13.848 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBfA6Ui0qAqUCicBPFTzUIamtbtaB2V2wf'], 'uuids': ['7bb249bc-6bc9-4c54-9b4b-69a99ee8405e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.065008", "end": "2021-05-12 11:55:13.814093", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBfA6Ui0qAqUCicBPFTzUIamtbtaB2V2wf"], "labels": [], "masters": [], "uuids": ["7bb249bc-6bc9-4c54-9b4b-69a99ee8405e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:12.749085", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:15.105 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBmKJJjPXTduJ7vGOAcfyDwo3EF2Md63Jp'], 'uuids': ['7b10ac71-d52f-4ecf-b40f-4999445b6836'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.082146", "end": "2021-05-12 11:55:15.075945", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBmKJJjPXTduJ7vGOAcfyDwo3EF2Md63Jp"], "labels": [], "masters": [], "uuids": ["7b10ac71-d52f-4ecf-b40f-4999445b6836"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:13.993799", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:16.350 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBis3ct2uODNacUUNB2u504l8LLg1JtDBz'], 'uuids': ['32cf12a0-58f5-4390-a2f6-cdf448ec6e48'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.068319", "end": "2021-05-12 11:55:16.323207", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBis3ct2uODNacUUNB2u504l8LLg1JtDBz"], "labels": [], "masters": [], "uuids": ["32cf12a0-58f5-4390-a2f6-cdf448ec6e48"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:15.254888", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:16.362 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C', 'scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C', 'scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C', 'scsi-35000cca3a8ce06d4', 'scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C', 'wwn-0x5000cca3a8ce06d4'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'Hitachi HUA72201', 'sas_address': None, 'sas_device_handle': None, 'serial': 'JPW9J0N20ZVK9C', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000cca3a8ce06d4', 'partitions': {'sda1': {'links': {'ids': ['ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1', 'scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1', 'scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1', 'scsi-35000cca3a8ce06d4-part1', 'scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1', 'wwn-0x5000cca3a8ce06d4-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C", "scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C", "scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C", "scsi-35000cca3a8ce06d4", "scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C", "wwn-0x5000cca3a8ce06d4"], "labels": [], "masters": [], "uuids": []}, "model": "Hitachi HUA72201", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-Hitachi_HUA 2021-05-12T11:55:16.363 INFO:teuthology.task.ansible.out:722010CLA330_JPW9J0N20ZVK9C-part1", "scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1", "scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1", "scsi-35000cca3a8ce06d4-part1", "scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1", "wwn-0x5000cca3a8ce06d4-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZVK9C", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000cca3a8ce06d4"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:17.597 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RB0r2lrfBeByHbaU2C74tIgOyW8iUQRDva'], 'uuids': ['4bd4c148-4c83-43d1-802d-0be2cd785975'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.055200", "end": "2021-05-12 11:55:17.564364", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RB0r2lrfBeByHbaU2C74tIgOyW8iUQRDva"], "labels": [], "masters": [], "uuids": ["4bd4c148-4c83-43d1-802d-0be2cd785975"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:16.509164", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T11:55:17.625 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:17 +0000 (0:00:08.129) 0:03:51.880 ********* 2021-05-12T11:55:18.022 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-05-12T11:55:18.024 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBDFzSGHcoMi7gSeZeFecoGNE3dTOTWtCJ'], 'uuids': ['0c2ecc44-086e-4bf5-a151-ce146ec2d80b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.142423", "end": "2021-05-12 11:55:17.990589", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBDFzSGHcoMi7gSeZeFecoGNE3dTOTWtCJ"], "labels": [], "masters": [], "uuids": ["0c2ecc44-086e-4bf5-a151-ce146ec2d80b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:17.848166", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139858 s, 825 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139858 s, 825 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:18.220 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-2swM1z-zau7-hgjc-OoT7-C5hZ-0eOT-JgX5lv', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UK400BGN', 'nvme-nvme.8086-50484654363230343030554b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037426", "end": "2021-05-12 11:55:18.197923", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2swM1z-zau7-hgjc-OoT7-C5hZ-0eOT-JgX5lv", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UK400BGN", "nvme-nvme.8086-50484654363230343030554b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:18.160497", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0347212 s, 3.3 GB/s", "std 2021-05-12T11:55:18.222 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0347212 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:18.541 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBfA6Ui0qAqUCicBPFTzUIamtbtaB2V2wf'], 'uuids': ['7bb249bc-6bc9-4c54-9b4b-69a99ee8405e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140742", "end": "2021-05-12 11:55:18.511017", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBfA6Ui0qAqUCicBPFTzUIamtbtaB2V2wf"], "labels": [], "masters": [], "uuids": ["7bb249bc-6bc9-4c54-9b4b-69a99ee8405e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:18.370275", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138118 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138118 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:18.843 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBmKJJjPXTduJ7vGOAcfyDwo3EF2Md63Jp'], 'uuids': ['7b10ac71-d52f-4ecf-b40f-4999445b6836'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142275", "end": "2021-05-12 11:55:18.820788", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBmKJJjPXTduJ7vGOAcfyDwo3EF2Md63Jp"], "labels": [], "masters": [], "uuids": ["7b10ac71-d52f-4ecf-b40f-4999445b6836"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:18.678513", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139776 s, 825 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139776 s, 825 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:20.161 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBis3ct2uODNacUUNB2u504l8LLg1JtDBz'], 'uuids': ['32cf12a0-58f5-4390-a2f6-cdf448ec6e48'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:01.143064", "end": "2021-05-12 11:55:20.131540", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RBis3ct2uODNacUUNB2u504l8LLg1JtDBz"], "labels": [], "masters": [], "uuids": ["32cf12a0-58f5-4390-a2f6-cdf448ec6e48"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:18.988476", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139413 s, 827 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139413 s, 827 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:20.173 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C', 'scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C', 'scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C', 'scsi-35000cca3a8ce06d4', 'scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C', 'wwn-0x5000cca3a8ce06d4'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'Hitachi HUA72201', 'sas_address': None, 'sas_device_handle': None, 'serial': 'JPW9J0N20ZVK9C', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000cca3a8ce06d4', 'partitions': {'sda1': {'links': {'ids': ['ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1', 'scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1', 'scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1', 'scsi-35000cca3a8ce06d4-part1', 'scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1', 'wwn-0x5000cca3a8ce06d4-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C", "scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C", "scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C", "scsi-35000cca3a8ce06d4", "scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C", "wwn-0x5000cca3a8ce06d4"], "labels": [], "masters": [], "uuids": []}, "model": "Hitachi HUA72201", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-Hitachi_HUA 2021-05-12T11:55:20.174 INFO:teuthology.task.ansible.out:722010CLA330_JPW9J0N20ZVK9C-part1", "scsi-0ATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1", "scsi-1ATA_Hitachi_HUA722010CLA330_JPW9J0N20ZVK9C-part1", "scsi-35000cca3a8ce06d4-part1", "scsi-SATA_Hitachi_HUA72201_JPW9J0N20ZVK9C-part1", "wwn-0x5000cca3a8ce06d4-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZVK9C", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000cca3a8ce06d4"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:20.493 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RB0r2lrfBeByHbaU2C74tIgOyW8iUQRDva'], 'uuids': ['4bd4c148-4c83-43d1-802d-0be2cd785975'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141921", "end": "2021-05-12 11:55:20.463438", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-d1IyrL0yJHT7qt1pnsQC3oZFRFLY53RB0r2lrfBeByHbaU2C74tIgOyW8iUQRDva"], "labels": [], "masters": [], "uuids": ["4bd4c148-4c83-43d1-802d-0be2cd785975"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 11:55:20.321517", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139246 s, 828 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139246 s, 828 MB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:20.521 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:20 +0000 (0:00:02.895) 0:03:54.776 ********* 2021-05-12T11:55:20.795 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-05-12T11:55:20.796 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.050787", "end": "2021-05-12 11:55:20.727489", "rc": 0, "start": "2021-05-12 11:55:20.676702", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:20.817 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:20 +0000 (0:00:00.295) 0:03:55.072 ********* 2021-05-12T11:55:21.042 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-05-12T11:55:21.043 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049239", "end": "2021-05-12 11:55:21.019175", "rc": 0, "start": "2021-05-12 11:55:20.969936", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:21.063 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.246) 0:03:55.319 ********* 2021-05-12T11:55:21.131 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-05-12T11:55:21.190 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.126) 0:03:55.445 ********* 2021-05-12T11:55:21.216 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-05-12T11:55:21.218 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.240 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.050) 0:03:55.495 ********* 2021-05-12T11:55:21.266 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-05-12T11:55:21.268 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.288 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.047) 0:03:55.543 ********* 2021-05-12T11:55:21.314 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-12T11:55:21.315 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.338 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.049) 0:03:55.593 ********* 2021-05-12T11:55:21.364 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-05-12T11:55:21.366 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.386 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.048) 0:03:55.641 ********* 2021-05-12T11:55:21.567 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-05-12T11:55:21.569 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T11:55:21.589 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.203) 0:03:55.845 ********* 2021-05-12T11:55:21.625 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-12T11:55:21.627 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278354944, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118739, 'block_used': 12976369, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118739, "block_size": 4096, "block_total": 240095108, "block_used": 12976369, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278354944, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.648 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.058) 0:03:55.903 ********* 2021-05-12T11:55:21.674 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-05-12T11:55:21.675 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.695 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.047) 0:03:55.951 ********* 2021-05-12T11:55:21.721 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:21.723 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.745 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.049) 0:03:56.000 ********* 2021-05-12T11:55:21.775 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:21.777 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.798 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.052) 0:03:56.053 ********* 2021-05-12T11:55:21.824 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.848 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.049) 0:03:56.103 ********* 2021-05-12T11:55:21.874 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-05-12T11:55:21.876 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.896 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.048) 0:03:56.152 ********* 2021-05-12T11:55:21.923 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-05-12T11:55:21.924 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:21.945 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:21 +0000 (0:00:00.048) 0:03:56.200 ********* 2021-05-12T11:55:22.697 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-05-12T11:55:22.698 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-05-12T11:55:22.725 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:22 +0000 (0:00:00.780) 0:03:56.981 ********* 2021-05-12T11:55:23.364 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-05-12T11:55:23.366 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:55:23.716 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:55:24.036 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:55:24.356 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:55:24.698 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T11:55:24.721 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:24 +0000 (0:00:01.995) 0:03:58.977 ********* 2021-05-12T11:55:24.913 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-12T11:55:24.915 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-05-12T11:55:24.935 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:24 +0000 (0:00:00.213) 0:03:59.191 ********* 2021-05-12T11:55:25.137 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-05-12T11:55:25.139 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:55:25.304 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:55:25.470 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:55:25.647 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T11:55:25.648 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:25.684 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:25 +0000 (0:00:00.748) 0:03:59.939 ********* 2021-05-12T11:55:25.710 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-05-12T11:55:25.712 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:25.731 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:25 +0000 (0:00:00.047) 0:03:59.986 ********* 2021-05-12T11:55:25.757 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-05-12T11:55:25.758 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:25.779 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:25 +0000 (0:00:00.048) 0:04:00.035 ********* 2021-05-12T11:55:25.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:55:25.996 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:25 +0000 (0:00:00.216) 0:04:00.251 ********* 2021-05-12T11:55:26.033 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-05-12T11:55:26.035 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-05-12T11:55:26.058 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:26 +0000 (0:00:00.062) 0:04:00.313 ********* 2021-05-12T11:55:26.621 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi090.front.sepia.ceph.com] => {"changed": true} 2021-05-12T11:55:26.644 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:26 +0000 (0:00:00.586) 0:04:00.900 ********* 2021-05-12T11:55:27.248 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-05-12T11:55:27.250 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-05-12T11:55:27.270 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.625) 0:04:01.525 ********* 2021-05-12T11:55:27.297 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-05-12T11:55:27.298 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.318 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.047) 0:04:01.573 ********* 2021-05-12T11:55:27.344 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-05-12T11:55:27.346 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.366 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.047) 0:04:01.621 ********* 2021-05-12T11:55:27.392 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T11:55:27.394 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.413 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.047) 0:04:01.669 ********* 2021-05-12T11:55:27.440 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T11:55:27.441 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.461 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.048) 0:04:01.717 ********* 2021-05-12T11:55:27.488 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-05-12T11:55:27.488 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.512 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.050) 0:04:01.768 ********* 2021-05-12T11:55:27.539 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-05-12T11:55:27.540 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.560 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.048) 0:04:01.816 ********* 2021-05-12T11:55:27.587 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-05-12T11:55:27.589 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.609 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.048) 0:04:01.864 ********* 2021-05-12T11:55:27.635 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-05-12T11:55:27.637 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.657 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.048) 0:04:01.912 ********* 2021-05-12T11:55:27.684 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-05-12T11:55:27.686 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.707 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.049) 0:04:01.962 ********* 2021-05-12T11:55:27.733 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-05-12T11:55:27.735 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.756 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.048) 0:04:02.011 ********* 2021-05-12T11:55:27.782 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-05-12T11:55:27.784 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.806 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.050) 0:04:02.062 ********* 2021-05-12T11:55:27.834 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-05-12T11:55:27.835 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.857 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.050) 0:04:02.112 ********* 2021-05-12T11:55:27.884 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-05-12T11:55:27.885 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.908 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.051) 0:04:02.163 ********* 2021-05-12T11:55:27.935 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-05-12T11:55:27.937 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:27.957 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:27 +0000 (0:00:00.049) 0:04:02.213 ********* 2021-05-12T11:55:27.984 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-05-12T11:55:27.986 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:28.007 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.049) 0:04:02.262 ********* 2021-05-12T11:55:28.456 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-05-12T11:55:28.458 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-05-12T11:55:28.480 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.473) 0:04:02.736 ********* 2021-05-12T11:55:28.778 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-05-12T11:55:28.779 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 11:50:49 UTC", "ActiveEnterTimestampMonotonic": "15732359", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service ntpdate.service systemd-journald.socket sysinit.target system.slice systemd-tmpfiles-setup.service basic.target ntpd.service -.mount tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 11:50:46 UTC", "AssertTimestampMonotonic": "13177539", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 11:50:46 UTC", "ConditionTimestampMonotonic": "13177511", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGro 2021-05-12T11:55:28.781 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "886", "ExecMainStartTimestamp": "Wed 2021-05-12 11:50:48 UTC", "ExecMainStartTimestampMonotonic": "15582125", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-05-12 11:50:46 UTC", "InactiveExitTimestampMonotonic": "13197359", "InvocationID": "5c6059df0d6d4e918541c3b3ad55230d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-05-12T11:55:28.783 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "886", "MemoryAccounting": "yes", "MemoryCurrent": "3395584", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-05-12T11:55:28.784 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 11:50:49 UTC", "StateChangeTimestampMonotonic": "15732359", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 11:50:48 UTC", "WatchdogTimestampMonotonic": "15582137", "WatchdogUSec": "0"}} 2021-05-12T11:55:28.802 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.321) 0:04:03.057 ********* 2021-05-12T11:55:28.828 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:28.830 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-05-12T11:55:28.849 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.047) 0:04:03.105 ********* 2021-05-12T11:55:28.886 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T11:55:28.888 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-05-12T11:55:28.908 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.058) 0:04:03.163 ********* 2021-05-12T11:55:28.935 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-05-12T11:55:28.937 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:28.957 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:28 +0000 (0:00:00.049) 0:04:03.212 ********* 2021-05-12T11:55:28.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-05-12T11:55:28.984 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:29.007 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:29 +0000 (0:00:00.049) 0:04:03.262 ********* 2021-05-12T11:55:29.033 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-05-12T11:55:29.035 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:29.054 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:29 +0000 (0:00:00.047) 0:04:03.310 ********* 2021-05-12T11:55:29.246 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-05-12T11:55:29.248 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-05-12T11:55:29.268 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:29 +0000 (0:00:00.213) 0:04:03.523 ********* 2021-05-12T11:55:29.726 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-05-12T11:55:29.728 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-05-12T11:55:29.748 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:29 +0000 (0:00:00.479) 0:04:04.003 ********* 2021-05-12T11:55:31.076 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-05-12T11:55:31.079 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-05-12T11:55:31.098 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:01.350) 0:04:05.353 ********* 2021-05-12T11:55:31.124 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-05-12T11:55:31.125 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:31.145 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:05.401 ********* 2021-05-12T11:55:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-05-12T11:55:31.173 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-05-12T11:55:31.193 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:05.448 ********* 2021-05-12T11:55:31.373 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-05-12T11:55:31.375 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-05-12T11:55:31.395 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.202) 0:04:05.650 ********* 2021-05-12T11:55:31.421 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-05-12T11:55:31.423 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:31.442 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:05.698 ********* 2021-05-12T11:55:31.468 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-05-12T11:55:31.470 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-05-12T11:55:31.489 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:05.745 ********* 2021-05-12T11:55:31.515 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-05-12T11:55:31.517 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:31.537 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:05.792 ********* 2021-05-12T11:55:31.728 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-05-12T11:55:31.731 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-05-12T11:55:31.750 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.213) 0:04:06.006 ********* 2021-05-12T11:55:31.932 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-05-12T11:55:32.028 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.202) 0:04:06.208 ********* TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 12 May 2021 11:55:31 +0000 (0:00:00.047) 0:04:06.256 ********* 2021-05-12T11:55:32.037 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-05-12T11:55:32.039 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.049 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.050 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.073 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.072) 0:04:06.328 ********* 2021-05-12T11:55:32.111 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-05-12T11:55:32.113 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi090.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.124 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.125 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.136 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.159 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.086) 0:04:06.414 ********* 2021-05-12T11:55:32.186 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.209 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.049) 0:04:06.464 ********* 2021-05-12T11:55:32.245 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-05-12T11:55:32.247 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.269 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.060) 0:04:06.524 ********* 2021-05-12T11:55:32.306 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-05-12T11:55:32.307 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.329 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.060) 0:04:06.585 ********* 2021-05-12T11:55:32.355 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-05-12T11:55:32.357 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.377 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.047) 0:04:06.632 ********* 2021-05-12T11:55:32.403 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-05-12T11:55:32.405 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.427 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.049) 0:04:06.682 ********* 2021-05-12T11:55:32.453 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T11:55:32.455 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.477 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.050) 0:04:06.733 ********* 2021-05-12T11:55:32.504 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-05-12T11:55:32.505 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.528 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.050) 0:04:06.783 ********* 2021-05-12T11:55:32.555 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T11:55:32.556 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:32.577 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:32 +0000 (0:00:00.048) 0:04:06.832 ********* 2021-05-12T11:55:32.790 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-12T11:55:32.792 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004756", "end": "2021-05-12 11:55:32.756753", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-05-12 11:55:32.751997", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:32.957 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002971", "end": "2021-05-12 11:55:32.927348", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-05-12 11:55:32.924377", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:33.123 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002744", "end": "2021-05-12 11:55:33.093035", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 11:55:33.090291", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:33.289 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003072", "end": "2021-05-12 11:55:33.262090", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-05-12 11:55:33.259018", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:33.312 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:33 +0000 (0:00:00.734) 0:04:07.567 ********* 2021-05-12T11:55:33.337 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-05-12T11:55:33.359 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:33 +0000 (0:00:00.047) 0:04:07.614 ********* 2021-05-12T11:55:33.550 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-05-12T11:55:33.552 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T11:55:33.619 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-05-12T11:55:33.638 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:33 +0000 (0:00:00.278) 0:04:07.893 ********* 2021-05-12T11:55:34.254 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi090.front.sepia.ceph.com] 2021-05-12T11:55:34.301 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:34 +0000 (0:00:00.663) 0:04:08.556 ********* 2021-05-12T11:55:34.327 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:55:34.329 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:34.348 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:34 +0000 (0:00:00.047) 0:04:08.604 ********* 2021-05-12T11:55:34.375 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:55:34.377 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-05-12T11:55:34.397 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:34 +0000 (0:00:00.048) 0:04:08.653 ********* 2021-05-12T11:55:34.435 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-05-12T11:55:34.436 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-05-12T11:55:34.457 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:34 +0000 (0:00:00.059) 0:04:08.712 ********* 2021-05-12T11:55:35.793 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-05-12T11:55:35.795 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T11:55:35.815 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:35 +0000 (0:00:01.357) 0:04:10.070 ********* 2021-05-12T11:55:35.852 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:55:35.853 INFO:teuthology.task.ansible.out:ok: [smithi090.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-05-12T11:55:35.874 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:35 +0000 (0:00:00.059) 0:04:10.129 ********* 2021-05-12T11:55:35.900 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T11:55:35.902 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:35.923 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:35 +0000 (0:00:00.049) 0:04:10.179 ********* 2021-05-12T11:55:36.136 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-12T11:55:36.138 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:55:36.323 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T11:55:36.349 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:36 +0000 (0:00:00.425) 0:04:10.604 ********* 2021-05-12T11:55:36.848 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-12T11:55:36.850 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820536.4196506-3287-12505520663863/source", "state": "file", "uid": 0} 2021-05-12T11:55:37.323 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620820536.8834276-3287-215996812964916/source", "state": "file", "uid": 0} 2021-05-12T11:55:37.356 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:37 +0000 (0:00:01.007) 0:04:11.611 ********* 2021-05-12T11:55:40.531 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-05-12T11:55:40.533 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-pthjhzj_-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-pthjhzj_-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-05-12T11:55:40.535 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-05-12T11:55:40.553 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:40 +0000 (0:00:03.197) 0:04:14.808 ********* 2021-05-12T11:55:40.580 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-05-12T11:55:40.581 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:40.603 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:40 +0000 (0:00:00.050) 0:04:14.858 ********* 2021-05-12T11:55:40.639 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-05-12T11:55:40.641 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:40.664 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:40 +0000 (0:00:00.060) 0:04:14.919 ********* 2021-05-12T11:55:41.041 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-05-12T11:55:41.043 INFO:teuthology.task.ansible.out:changed: [smithi090.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.171629", "end": "2021-05-12 11:55:41.020609", "rc": 0, "start": "2021-05-12 11:55:40.848980", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T11:55:41.063 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 11:55:41 +0000 (0:00:00.399) 0:04:15.318 ********* 2021-05-12T11:55:41.089 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-05-12T11:55:41.091 INFO:teuthology.task.ansible.out:skipping: [smithi090.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T11:55:41.155 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-05-12T11:55:41.156 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-05-12T11:55:41.158 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi090.front.sepia.ceph.com : ok=127 changed=42 unreachable=0 failed=0 skipped=185 rescued=0 ignored=3 Wednesday 12 May 2021 11:55:41 +0000 (0:00:00.094) 0:04:15.413 ********* =============================================================================== 2021-05-12T11:55:41.158 INFO:teuthology.task.ansible.out:common : nrpe - Ensure SELinux policy is up to date -------------------- 38.58s 2021-05-12T11:55:41.159 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 28.68s users : Update authorized_keys using the keys repo --------------------- 28.27s common : nrpe - Load SELinux policy package ---------------------------- 16.67s common : nrpe - Remove SELinux policy package -------------------------- 16.46s Remove revoked users --------------------------------------------------- 10.62s testnode : Zap all non-root disks --------------------------------------- 8.13s testnode : Install packages --------------------------------------------- 7.08s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 6.19s common : Install Kerberos Packages (RedHat) ----------------------------- 5.73s container-host : Install registries-conf-ctl ---------------------------- 3.20s testnode : Ensure ceph packages are not present. ------------------------ 2.91s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.90s testnode : Ensure ceph dependency packages are not present. ------------- 2.68s testnode : Rebuild rpmdb ------------------------------------------------ 2.64s testnode : Create logical volume(s) ------------------------------------- 2.00s users : Update authorized_keys for each user with literal keys ---------- 1.69s Gathering Facts --------------------------------------------------------- 1.68s 2021-05-12T11:55:41.159 INFO:teuthology.task.ansible.out:Gathering Facts ----------------------------------------- 2021-05-12T11:55:41.161 INFO:teuthology.task.ansible.out:---------------- 1.55s testnode : Remove packages ---------------------------------------------- 1.52s 2021-05-12T11:55:41.487 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2021-05-12T11:55:41.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T11:55:41.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T11:55:41.562 DEBUG:teuthology.orchestra.run.smithi090:> true 2021-05-12T11:55:41.654 INFO:teuthology.run_tasks:Running task clock... 2021-05-12T11:55:41.666 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-05-12T11:55:41.666 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-05-12T11:55:41.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-05-12T11:55:41.725 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-05-12T11:55:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-05-12T11:55:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2021-05-12T11:55:41.782 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2021-05-12T11:55:41.796 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-05-12T11:55:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-05-12T11:55:41.870 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2021-05-12T11:55:41.873 INFO:teuthology.orchestra.run.smithi090.stdout:210 Number of sources = 4 2021-05-12T11:55:41.873 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-05-12T11:55:41.873 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2021-05-12T11:55:41.874 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:55:41.874 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:55:41.874 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:55:41.875 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T11:55:41.875 INFO:teuthology.run_tasks:Running task install... 2021-05-12T11:55:41.886 DEBUG:teuthology.task.install:project ceph 2021-05-12T11:55:41.887 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740'}} 2021-05-12T11:55:41.887 DEBUG:teuthology.task.install:config {'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740'} 2021-05-12T11:55:41.888 INFO:teuthology.task.install:Using flavor: basic 2021-05-12T11:55:41.895 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-05-12T11:55:41.896 INFO:teuthology.task.install:extra packages: [] 2021-05-12T11:55:41.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2021-05-12T11:55:41.945 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS 2021-05-12T11:55:41.946 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T11:55:41.946 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '36c2cf2fb22a9790b2c097e158985773b35f1740', 'tag': None, 'wait_for_package': False} 2021-05-12T11:55:41.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2021-05-12T11:55:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS 2021-05-12T11:55:42.018 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T11:55:42.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:55:42.170 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/ 2021-05-12T11:55:42.170 INFO:teuthology.task.install.rpm:Package version is 17.0.0-4087.g36c2cf2f 2021-05-12T11:55:42.389 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1746/36c2cf2fb22a9790b2c097e158985773b35f1740/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-05-12T11:55:42.390 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:55:42.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-05-12T11:55:42.420 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-05-12T11:55:42.421 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/36c2cf2fb22a9790b2c097e158985773b35f1740/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-05-12T11:55:42.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-05-12T11:55:42.584 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-05-12T11:55:42.654 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2021-05-12T11:55:42.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2021-05-12T11:55:43.013 INFO:teuthology.orchestra.run.smithi090.stdout:72 files removed 2021-05-12T11:55:43.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2021-05-12T11:55:43.697 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 317 kB/s | 77 kB 00:00 2021-05-12T11:55:43.928 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 77 kB/s | 16 kB 00:00 2021-05-12T11:55:44.151 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 7.2 kB/s | 1.5 kB 00:00 2021-05-12T11:55:44.407 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - AppStream 26 MB/s | 6.3 MB 00:00 2021-05-12T11:55:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - Base 13 MB/s | 2.3 MB 00:00 2021-05-12T11:55:46.022 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-8 - Extras 138 kB/s | 9.6 kB 00:00 2021-05-12T11:55:52.031 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - AppStream 1.1 MB/s | 6.3 MB 00:05 2021-05-12T11:55:55.667 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - BaseOS 903 kB/s | 2.3 MB 00:02 2021-05-12T11:55:56.436 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Linux 8 - Extras 24 kB/s | 9.6 kB 00:00 2021-05-12T11:55:57.414 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 10 MB/s | 9.4 MB 00:00 2021-05-12T11:55:59.409 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 5.0 MB/s | 24 kB 00:00 2021-05-12T11:56:01.020 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:56:01.037 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:56:01.038 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:01.038 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:56:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 9.9 M 2021-05-12T11:56:01.039 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2021-05-12T11:56:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.3 M 2021-05-12T11:56:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:56:01.039 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:56:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 5.4 M 2021-05-12T11:56:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 19 M 2021-05-12T11:56:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 23 k 2021-05-12T11:56:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-05-12T11:56:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-05-12T11:56:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-05-12T11:56:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 756 k 2021-05-12T11:56:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-05-12T11:56:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-05-12T11:56:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 458 k 2021-05-12T11:56:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-05-12T11:56:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:56:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-05-12T11:56:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-05-12T11:56:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse 2021-05-12T11:56:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 42 k 2021-05-12T11:56:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 77 k 2021-05-12T11:56:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 177 k 2021-05-12T11:56:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-05-12T11:56:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 364 k 2021-05-12T11:56:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 345 k 2021-05-12T11:56:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 109 k 2021-05-12T11:56:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:01.045 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:56:01.045 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:01.045 INFO:teuthology.orchestra.run.smithi090.stdout:Install 22 Packages 2021-05-12T11:56:01.045 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2021-05-12T11:56:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:01.046 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 49 M 2021-05-12T11:56:01.047 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:56:01.687 INFO:teuthology.orchestra.run.smithi090.stdout:(1/24): ceph-base-17.0.0-4087.g36c2cf2f.el8.x86 8.4 MB/s | 5.4 MB 00:00 2021-05-12T11:56:01.730 INFO:teuthology.orchestra.run.smithi090.stdout:(2/24): ceph-selinux-17.0.0-4087.g36c2cf2f.el8. 549 kB/s | 23 kB 00:00 2021-05-12T11:56:01.831 INFO:teuthology.orchestra.run.smithi090.stdout:(3/24): libcephfs2-17.0.0-4087.g36c2cf2f.el8.x8 7.4 MB/s | 756 kB 00:00 2021-05-12T11:56:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:(4/24): libradosstriper1-17.0.0-4087.g36c2cf2f. 8.2 MB/s | 458 kB 00:00 2021-05-12T11:56:02.257 INFO:teuthology.orchestra.run.smithi090.stdout:(5/24): librgw2-17.0.0-4087.g36c2cf2f.el8.x86_6 10 MB/s | 3.8 MB 00:00 2021-05-12T11:56:02.301 INFO:teuthology.orchestra.run.smithi090.stdout:(6/24): python3-ceph-argparse-17.0.0-4087.g36c2 951 kB/s | 42 kB 00:00 2021-05-12T11:56:02.345 INFO:teuthology.orchestra.run.smithi090.stdout:(7/24): python3-ceph-common-17.0.0-4087.g36c2cf 1.7 MB/s | 77 kB 00:00 2021-05-12T11:56:02.393 INFO:teuthology.orchestra.run.smithi090.stdout:(8/24): python3-cephfs-17.0.0-4087.g36c2cf2f.el 3.6 MB/s | 177 kB 00:00 2021-05-12T11:56:02.448 INFO:teuthology.orchestra.run.smithi090.stdout:(9/24): python3-rados-17.0.0-4087.g36c2cf2f.el8 6.4 MB/s | 364 kB 00:00 2021-05-12T11:56:02.503 INFO:teuthology.orchestra.run.smithi090.stdout:(10/24): ceph-radosgw-17.0.0-4087.g36c2cf2f.el8 6.8 MB/s | 9.9 MB 00:01 2021-05-12T11:56:02.506 INFO:teuthology.orchestra.run.smithi090.stdout:(11/24): python3-rbd-17.0.0-4087.g36c2cf2f.el8. 5.8 MB/s | 345 kB 00:00 2021-05-12T11:56:02.534 INFO:teuthology.orchestra.run.smithi090.stdout:(12/24): python3-rgw-17.0.0-4087.g36c2cf2f.el8. 3.4 MB/s | 109 kB 00:00 2021-05-12T11:56:02.544 INFO:teuthology.orchestra.run.smithi090.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.2 MB/s | 353 kB 00:00 2021-05-12T11:56:02.546 INFO:teuthology.orchestra.run.smithi090.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 30 MB/s | 44 kB 00:00 2021-05-12T11:56:02.547 INFO:teuthology.orchestra.run.smithi090.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 31 MB/s | 47 kB 00:00 2021-05-12T11:56:02.566 INFO:teuthology.orchestra.run.smithi090.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.1 MB/s | 259 kB 00:00 2021-05-12T11:56:02.722 INFO:teuthology.orchestra.run.smithi090.stdout:(17/24): ceph-common-17.0.0-4087.g36c2cf2f.el8. 11 MB/s | 19 MB 00:01 2021-05-12T11:56:02.723 INFO:teuthology.orchestra.run.smithi090.stdout:(18/24): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2021-05-12T11:56:02.746 INFO:teuthology.orchestra.run.smithi090.stdout:(19/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2021-05-12T11:56:02.750 INFO:teuthology.orchestra.run.smithi090.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2021-05-12T11:56:02.768 INFO:teuthology.orchestra.run.smithi090.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 3.2 MB/s | 75 kB 00:00 2021-05-12T11:56:02.861 INFO:teuthology.orchestra.run.smithi090.stdout:(22/24): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-05-12T11:56:03.020 INFO:teuthology.orchestra.run.smithi090.stdout:(23/24): librados2-17.0.0-4087.g36c2cf2f.el8.x8 12 MB/s | 3.3 MB 00:00 2021-05-12T11:56:03.057 INFO:teuthology.orchestra.run.smithi090.stdout:(24/24): librbd1-17.0.0-4087.g36c2cf2f.el8.x86_ 13 MB/s | 3.8 MB 00:00 2021-05-12T11:56:03.061 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:56:03.061 INFO:teuthology.orchestra.run.smithi090.stdout:Total 24 MB/s | 49 MB 00:02 2021-05-12T11:56:03.064 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:56:03.110 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:56:03.111 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:56:03.484 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:56:03.487 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:56:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:56:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-05-12T11:56:04.811 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-05-12T11:56:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-12T11:56:05.358 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-12T11:56:05.553 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/26 2021-05-12T11:56:05.650 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/26 2021-05-12T11:56:05.898 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 5/26 2021-05-12T11:56:06.111 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/26 2021-05-12T11:56:06.573 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/26 2021-05-12T11:56:06.728 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:56:06.789 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:56:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-05-12T11:56:07.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T11:56:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T11:56:07.792 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:56:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:56:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-05-12T11:56:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8. 12/26 2021-05-12T11:56:08.323 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 13/26 2021-05-12T11:56:08.492 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 14/26 2021-05-12T11:56:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 15/26 2021-05-12T11:56:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 16/26 2021-05-12T11:56:08.947 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 16/26 2021-05-12T11:56:09.130 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-05-12T11:56:09.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-05-12T11:56:09.423 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-05-12T11:56:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x8 20/26 2021-05-12T11:56:13.568 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:56:13.778 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:56:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 21/26 2021-05-12T11:56:14.780 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 22/26 2021-05-12T11:56:14.871 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 22/26 2021-05-12T11:56:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:56:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:56:32.019 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2021-05-12T11:56:32.019 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2021-05-12T11:56:32.020 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2021-05-12T11:56:32.020 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /run 2021-05-12T11:56:32.020 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2021-05-12T11:56:32.020 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2021-05-12T11:56:32.020 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2021-05-12T11:56:32.021 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2021-05-12T11:56:32.021 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2021-05-12T11:56:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:33.246 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 24/26 2021-05-12T11:56:33.326 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 24/26 2021-05-12T11:56:33.327 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T11:56:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T11:56:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:56:34.268 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:56:34.268 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/26 2021-05-12T11:56:34.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/26 2021-05-12T11:56:34.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/26 2021-05-12T11:56:34.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/26 2021-05-12T11:56:34.269 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 5/26 2021-05-12T11:56:34.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_6 6/26 2021-05-12T11:56:34.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/26 2021-05-12T11:56:34.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8. 8/26 2021-05-12T11:56:34.270 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x8 9/26 2021-05-12T11:56:34.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 10/26 2021-05-12T11:56:34.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 11/26 2021-05-12T11:56:34.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 12/26 2021-05-12T11:56:34.271 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 13/26 2021-05-12T11:56:34.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-05-12T11:56:34.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-05-12T11:56:34.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-05-12T11:56:34.272 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-05-12T11:56:34.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-05-12T11:56:34.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-05-12T11:56:34.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-05-12T11:56:34.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-05-12T11:56:34.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-05-12T11:56:34.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 23/26 2021-05-12T11:56:34.274 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-05-12T11:56:34.275 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 25/26 2021-05-12T11:56:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T11:56:35.190 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:56:35.466 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:35.467 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2021-05-12T11:56:35.467 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.467 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.467 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:35.468 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:56:35.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.469 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.469 INFO:teuthology.orchestra.run.smithi090.stdout: fmt-6.2.1-1.el8.x86_64 2021-05-12T11:56:35.469 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-05-12T11:56:35.469 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb-1.22-1.el8.x86_64 2021-05-12T11:56:35.470 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.470 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.2-3.el8.x86_64 2021-05-12T11:56:35.470 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-05-12T11:56:35.470 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.471 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-05-12T11:56:35.471 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.471 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.3.1-3.el8.x86_64 2021-05-12T11:56:35.471 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-05-12T11:56:35.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.472 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-05-12T11:56:35.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.473 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:56:35.473 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:35.474 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:56:35.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2021-05-12T11:56:36.135 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:37 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:56:36.868 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:56:36.869 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:36.869 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:56:36.869 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:36.870 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:56:36.870 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 43 M 2021-05-12T11:56:36.870 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:56:36.870 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-05-12T11:56:36.871 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 159 k 2021-05-12T11:56:36.871 INFO:teuthology.orchestra.run.smithi090.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-05-12T11:56:36.871 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-05-12T11:56:36.871 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-05-12T11:56:36.871 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-05-12T11:56:36.872 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:36.872 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:56:36.872 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:56:36.872 INFO:teuthology.orchestra.run.smithi090.stdout:Install 7 Packages 2021-05-12T11:56:36.872 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:36.873 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 44 M 2021-05-12T11:56:36.873 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 243 M 2021-05-12T11:56:36.874 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:56:36.972 INFO:teuthology.orchestra.run.smithi090.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 1.6 MB/s | 161 kB 00:00 2021-05-12T11:56:36.975 INFO:teuthology.orchestra.run.smithi090.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 58 MB/s | 187 kB 00:00 2021-05-12T11:56:36.980 INFO:teuthology.orchestra.run.smithi090.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 67 MB/s | 302 kB 00:00 2021-05-12T11:56:36.984 INFO:teuthology.orchestra.run.smithi090.stdout:(4/7): libxslt-1.1.32-5.el8.x86_64.rpm 63 MB/s | 250 kB 00:00 2021-05-12T11:56:37.048 INFO:teuthology.orchestra.run.smithi090.stdout:(5/7): libcephsqlite-17.0.0-4087.g36c2cf2f.el8. 903 kB/s | 159 kB 00:00 2021-05-12T11:56:37.174 INFO:teuthology.orchestra.run.smithi090.stdout:(6/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 363 kB/s | 69 kB 00:00 2021-05-12T11:56:38.997 INFO:teuthology.orchestra.run.smithi090.stdout:(7/7): ceph-test-17.0.0-4087.g36c2cf2f.el8.x86_ 20 MB/s | 43 MB 00:02 2021-05-12T11:56:39.001 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:56:39.002 INFO:teuthology.orchestra.run.smithi090.stdout:Total 21 MB/s | 44 MB 00:02 2021-05-12T11:56:39.005 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:56:39.030 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:56:39.031 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:56:39.549 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:56:39.607 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:56:40.484 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:56:40.662 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-05-12T11:56:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-05-12T11:56:40.988 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-05-12T11:56:41.160 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T11:56:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T11:56:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-05-12T11:56:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/7 2021-05-12T11:56:46.592 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 6/7 2021-05-12T11:56:47.191 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/7 2021-05-12T11:56:47.428 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 7/7 2021-05-12T11:56:47.428 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/7 2021-05-12T11:56:47.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/7 2021-05-12T11:56:47.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-05-12T11:56:47.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T11:56:47.429 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-05-12T11:56:47.430 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 6/7 2021-05-12T11:56:47.836 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-05-12T11:56:47.837 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:56:48.062 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:48.062 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:56:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:17.0.0-4087.g36c2cf2f.el8.x86_64 jq-1.5-12.el8.x86_64 2021-05-12T11:56:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:17.0.0-4087.g36c2cf2f.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-05-12T11:56:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-05-12T11:56:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-05-12T11:56:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:48.064 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:56:48.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2021-05-12T11:56:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:49 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:56:49.439 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:56:49.443 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2021-05-12T11:56:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:56:49.444 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2021-05-12T11:56:49.444 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:56:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 6.4 k 2021-05-12T11:56:49.444 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:56:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 2.2 M 2021-05-12T11:56:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 1.5 M 2021-05-12T11:56:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 204 k 2021-05-12T11:56:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.8 M 2021-05-12T11:56:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 16 M 2021-05-12T11:56:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-05-12T11:56:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-05-12T11:56:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-05-12T11:56:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-05-12T11:56:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-05-12T11:56:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-05-12T11:56:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-05-12T11:56:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-05-12T11:56:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-05-12T11:56:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-05-12T11:56:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-05-12T11:56:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-05-12T11:56:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-05-12T11:56:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-05-12T11:56:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-05-12T11:56:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-05-12T11:56:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-05-12T11:56:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-05-12T11:56:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-05-12T11:56:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-05-12T11:56:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-05-12T11:56:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-05-12T11:56:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-05-12T11:56:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-05-12T11:56:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-05-12T11:56:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-05-12T11:56:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-05-12T11:56:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-05-12T11:56:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-05-12T11:56:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-05-12T11:56:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-05-12T11:56:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-05-12T11:56:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-05-12T11:56:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-05-12T11:56:49.453 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2021-05-12T11:56:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-05-12T11:56:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-05-12T11:56:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:49.454 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:56:49.454 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2021-05-12T11:56:49.455 INFO:teuthology.orchestra.run.smithi090.stdout:Install 42 Packages 2021-05-12T11:56:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:56:49.455 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2021-05-12T11:56:49.456 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 126 M 2021-05-12T11:56:49.456 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:56:49.576 INFO:teuthology.orchestra.run.smithi090.stdout:(1/42): ceph-17.0.0-4087.g36c2cf2f.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2021-05-12T11:56:49.965 INFO:teuthology.orchestra.run.smithi090.stdout:(2/42): ceph-mgr-17.0.0-4087.g36c2cf2f.el8.x86_ 2.8 MB/s | 1.5 MB 00:00 2021-05-12T11:56:50.072 INFO:teuthology.orchestra.run.smithi090.stdout:(3/42): ceph-mds-17.0.0-4087.g36c2cf2f.el8.x86_ 3.6 MB/s | 2.2 MB 00:00 2021-05-12T11:56:50.112 INFO:teuthology.orchestra.run.smithi090.stdout:(4/42): ceph-mgr-modules-core-17.0.0-4087.g36c2 5.0 MB/s | 204 kB 00:00 2021-05-12T11:56:50.146 INFO:teuthology.orchestra.run.smithi090.stdout:(5/42): python3-html5lib-0.999999999-6.el8.noar 6.1 MB/s | 214 kB 00:00 2021-05-12T11:56:50.167 INFO:teuthology.orchestra.run.smithi090.stdout:(6/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 75 MB/s | 1.5 MB 00:00 2021-05-12T11:56:50.170 INFO:teuthology.orchestra.run.smithi090.stdout:(7/42): python3-mako-1.0.6-13.el8.noarch.rpm 55 MB/s | 157 kB 00:00 2021-05-12T11:56:50.171 INFO:teuthology.orchestra.run.smithi090.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 25 MB/s | 39 kB 00:00 2021-05-12T11:56:50.175 INFO:teuthology.orchestra.run.smithi090.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 25 MB/s | 103 kB 00:00 2021-05-12T11:56:50.177 INFO:teuthology.orchestra.run.smithi090.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2021-05-12T11:56:50.178 INFO:teuthology.orchestra.run.smithi090.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 19 MB/s | 27 kB 00:00 2021-05-12T11:56:50.186 INFO:teuthology.orchestra.run.smithi090.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 62 MB/s | 457 kB 00:00 2021-05-12T11:56:50.189 INFO:teuthology.orchestra.run.smithi090.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 57 MB/s | 181 kB 00:00 2021-05-12T11:56:50.193 INFO:teuthology.orchestra.run.smithi090.stdout:(14/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 67 MB/s | 237 kB 00:00 2021-05-12T11:56:50.196 INFO:teuthology.orchestra.run.smithi090.stdout:(15/42): python3-chardet-3.0.4-7.el8.noarch.rpm 61 MB/s | 195 kB 00:00 2021-05-12T11:56:50.204 INFO:teuthology.orchestra.run.smithi090.stdout:(16/42): python3-cryptography-2.3-3.el8.x86_64. 65 MB/s | 510 kB 00:00 2021-05-12T11:56:50.206 INFO:teuthology.orchestra.run.smithi090.stdout:(17/42): python3-idna-2.5-5.el8.noarch.rpm 41 MB/s | 97 kB 00:00 2021-05-12T11:56:50.209 INFO:teuthology.orchestra.run.smithi090.stdout:(18/42): python3-pycparser-2.14-14.el8.noarch.r 48 MB/s | 109 kB 00:00 2021-05-12T11:56:50.210 INFO:teuthology.orchestra.run.smithi090.stdout:(19/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 25 MB/s | 34 kB 00:00 2021-05-12T11:56:50.213 INFO:teuthology.orchestra.run.smithi090.stdout:(20/42): python3-requests-2.20.0-2.1.el8_1.noar 54 MB/s | 123 kB 00:00 2021-05-12T11:56:50.216 INFO:teuthology.orchestra.run.smithi090.stdout:(21/42): python3-urllib3-1.24.2-4.el8.noarch.rp 62 MB/s | 176 kB 00:00 2021-05-12T11:56:50.343 INFO:teuthology.orchestra.run.smithi090.stdout:(22/42): ceph-mon-17.0.0-4087.g36c2cf2f.el8.x86 5.0 MB/s | 3.8 MB 00:00 2021-05-12T11:56:50.618 INFO:teuthology.orchestra.run.smithi090.stdout:(23/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 108 kB/s | 44 kB 00:00 2021-05-12T11:56:50.755 INFO:teuthology.orchestra.run.smithi090.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 450 kB/s | 185 kB 00:00 2021-05-12T11:56:50.804 INFO:teuthology.orchestra.run.smithi090.stdout:(25/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 927 kB/s | 173 kB 00:00 2021-05-12T11:56:50.865 INFO:teuthology.orchestra.run.smithi090.stdout:(26/42): python3-cssselect-0.9.2-10.el8.noarch. 660 kB/s | 40 kB 00:00 2021-05-12T11:56:50.916 INFO:teuthology.orchestra.run.smithi090.stdout:(27/42): python3-jaraco-6.2-6.el8.noarch.rpm 218 kB/s | 11 kB 00:00 2021-05-12T11:56:50.949 INFO:teuthology.orchestra.run.smithi090.stdout:(28/42): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2021-05-12T11:56:50.967 INFO:teuthology.orchestra.run.smithi090.stdout:(29/42): python3-jaraco-functools-2.0-4.el8.noa 342 kB/s | 18 kB 00:00 2021-05-12T11:56:51.001 INFO:teuthology.orchestra.run.smithi090.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 941 kB/s | 49 kB 00:00 2021-05-12T11:56:51.023 INFO:teuthology.orchestra.run.smithi090.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2021-05-12T11:56:51.074 INFO:teuthology.orchestra.run.smithi090.stdout:(32/42): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2021-05-12T11:56:51.106 INFO:teuthology.orchestra.run.smithi090.stdout:(33/42): python3-pecan-1.3.2-9.el8.noarch.rpm 2.6 MB/s | 283 kB 00:00 2021-05-12T11:56:51.123 INFO:teuthology.orchestra.run.smithi090.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 383 kB/s | 19 kB 00:00 2021-05-12T11:56:51.155 INFO:teuthology.orchestra.run.smithi090.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 492 kB/s | 24 kB 00:00 2021-05-12T11:56:51.171 INFO:teuthology.orchestra.run.smithi090.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 600 kB/s | 29 kB 00:00 2021-05-12T11:56:51.203 INFO:teuthology.orchestra.run.smithi090.stdout:(37/42): python3-trustme-0.6.0-4.el8.noarch.rpm 572 kB/s | 27 kB 00:00 2021-05-12T11:56:51.297 INFO:teuthology.orchestra.run.smithi090.stdout:(38/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.6 MB/s | 251 kB 00:00 2021-05-12T11:56:51.316 INFO:teuthology.orchestra.run.smithi090.stdout:(39/42): python3-waitress-1.2.1-2.el8.1.noarch. 1.6 MB/s | 241 kB 00:00 2021-05-12T11:56:51.342 INFO:teuthology.orchestra.run.smithi090.stdout:(40/42): python3-webtest-2.0.33-1.el8.noarch.rp 1.9 MB/s | 85 kB 00:00 2021-05-12T11:56:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 526 kB/s | 23 kB 00:00 2021-05-12T11:56:52.120 INFO:teuthology.orchestra.run.smithi090.stdout:(42/42): ceph-osd-17.0.0-4087.g36c2cf2f.el8.x86 7.5 MB/s | 16 MB 00:02 2021-05-12T11:56:52.123 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:56:52.124 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 30 MB 00:02 2021-05-12T11:56:52.137 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:56:52.180 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:56:52.181 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:56:52.699 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:56:52.703 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:56:54.151 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:56:54.354 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-05-12T11:56:54.560 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-05-12T11:56:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-05-12T11:56:54.966 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-05-12T11:56:55.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-05-12T11:56:55.331 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-05-12T11:56:55.503 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-05-12T11:56:55.696 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-05-12T11:56:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-05-12T11:56:56.013 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-05-12T11:56:56.192 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-05-12T11:56:56.433 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-05-12T11:56:56.658 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-05-12T11:56:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-05-12T11:56:57.052 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-05-12T11:56:57.301 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-05-12T11:56:57.539 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-12T11:56:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-05-12T11:56:58.045 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-05-12T11:56:58.254 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-05-12T11:56:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-05-12T11:56:58.635 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-05-12T11:56:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-05-12T11:56:59.121 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-05-12T11:56:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-05-12T11:56:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-05-12T11:56:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-05-12T11:57:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-05-12T11:57:00.332 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-05-12T11:57:00.487 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-05-12T11:57:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-05-12T11:57:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-05-12T11:57:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-05-12T11:57:01.397 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-05-12T11:57:01.707 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-05-12T11:57:02.057 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-05-12T11:57:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8. 37/42 2021-05-12T11:57:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 38/42 2021-05-12T11:57:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 38/42 2021-05-12T11:57:04.286 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 39/42 2021-05-12T11:57:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 39/42 2021-05-12T11:57:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 40/42 2021-05-12T11:57:05.129 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 40/42 2021-05-12T11:57:05.326 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 41/42 2021-05-12T11:57:05.345 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 41/42 2021-05-12T11:57:05.475 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 42/42 2021-05-12T11:57:05.846 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 42/42 2021-05-12T11:57:05.846 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/42 2021-05-12T11:57:05.846 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/42 2021-05-12T11:57:05.847 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 3/42 2021-05-12T11:57:05.847 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 4/42 2021-05-12T11:57:05.847 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 5/42 2021-05-12T11:57:05.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8. 6/42 2021-05-12T11:57:05.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-05-12T11:57:05.848 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-05-12T11:57:05.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-05-12T11:57:05.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-05-12T11:57:05.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-05-12T11:57:05.849 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-05-12T11:57:05.850 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-05-12T11:57:05.850 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-05-12T11:57:05.850 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-05-12T11:57:05.850 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-05-12T11:57:05.851 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-12T11:57:05.851 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-05-12T11:57:05.851 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-05-12T11:57:05.851 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-05-12T11:57:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-05-12T11:57:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-05-12T11:57:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-05-12T11:57:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-05-12T11:57:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-05-12T11:57:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-05-12T11:57:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-05-12T11:57:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-05-12T11:57:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-05-12T11:57:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-05-12T11:57:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-05-12T11:57:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-05-12T11:57:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-05-12T11:57:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-05-12T11:57:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-05-12T11:57:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-05-12T11:57:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-05-12T11:57:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-05-12T11:57:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-05-12T11:57:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-05-12T11:57:05.858 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-05-12T11:57:06.674 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-05-12T11:57:06.674 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:06.908 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-05-12T11:57:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-05-12T11:57:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-05-12T11:57:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-05-12T11:57:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-05-12T11:57:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-05-12T11:57:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-05-12T11:57:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-05-12T11:57:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-05-12T11:57:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-05-12T11:57:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-idna-2.5-5.el8.noarch 2021-05-12T11:57:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-6.2-6.el8.noarch 2021-05-12T11:57:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-05-12T11:57:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-05-12T11:57:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-05-12T11:57:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.0.6-13.el8.noarch 2021-05-12T11:57:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-05-12T11:57:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-05-12T11:57:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-05-12T11:57:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-2.6-1.el8.noarch 2021-05-12T11:57:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-05-12T11:57:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.14-14.el8.noarch 2021-05-12T11:57:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-05-12T11:57:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2017.2-9.el8.noarch 2021-05-12T11:57:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-05-12T11:57:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-05-12T11:57:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-05-12T11:57:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-05-12T11:57:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-05-12T11:57:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-05-12T11:57:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-05-12T11:57:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-05-12T11:57:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-05-12T11:57:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-05-12T11:57:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-05-12T11:57:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-05-12T11:57:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:06.918 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:07.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2021-05-12T11:57:07.657 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:08 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:08.138 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:57:08.308 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:08.309 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:57:08.309 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:08.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2021-05-12T11:57:08.906 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:09 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:09.555 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:09.556 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:09.557 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:09.557 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:09.557 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:09.557 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 67 k 2021-05-12T11:57:09.558 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:09.558 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:09.558 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:09.558 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:57:09.559 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:09.559 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 67 k 2021-05-12T11:57:09.559 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 296 k 2021-05-12T11:57:09.559 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:09.703 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-17.0.0-4087.g36c2cf2f.el8.noarch.rpm 459 kB/s | 67 kB 00:00 2021-05-12T11:57:09.704 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:09.704 INFO:teuthology.orchestra.run.smithi090.stdout:Total 455 kB/s | 67 kB 00:00 2021-05-12T11:57:09.704 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:09.711 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:09.712 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:09.713 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:09.716 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:09.759 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:11.793 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:57:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:57:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:57:12.472 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/1 2021-05-12T11:57:12.472 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:12.681 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:12.681 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:12.681 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:12.682 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:12.682 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:12.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2021-05-12T11:57:13.390 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:14 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:14.720 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:14.721 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2021-05-12T11:57:14.721 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:14.722 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 167 k 2021-05-12T11:57:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:14.722 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:14.723 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:14.723 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:57:14.723 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:14.723 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 167 k 2021-05-12T11:57:14.723 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 633 k 2021-05-12T11:57:14.724 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:15.090 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-17.0.0-4087.g36c2cf 453 kB/s | 167 kB 00:00 2021-05-12T11:57:15.092 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:15.092 INFO:teuthology.orchestra.run.smithi090.stdout:Total 451 kB/s | 167 kB 00:00 2021-05-12T11:57:15.092 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:15.107 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:15.107 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:15.158 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:15.161 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:15.833 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:57:16.992 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:57:17.218 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f. 1/1 2021-05-12T11:57:17.218 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:17.469 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:17.469 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:17.469 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:17.469 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:17.470 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:17.654 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2021-05-12T11:57:18.188 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:19 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:18.671 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:57:18.838 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:18.839 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:57:18.839 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:18.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2021-05-12T11:57:19.443 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:20 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:20.126 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:20.126 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:20.127 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:20.127 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:20.127 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:20.127 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 2.3 M 2021-05-12T11:57:20.128 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:57:20.128 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 24 k 2021-05-12T11:57:20.128 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 8.9 k 2021-05-12T11:57:20.128 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-05-12T11:57:20.128 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-05-12T11:57:20.129 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-05-12T11:57:20.129 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:20.129 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:20.129 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:20.130 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2021-05-12T11:57:20.130 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:20.131 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.6 M 2021-05-12T11:57:20.131 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 15 M 2021-05-12T11:57:20.131 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:20.310 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-prometheus-alerts-17.0.0-4087.g36c2 49 kB/s | 8.9 kB 00:00 2021-05-12T11:57:20.341 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-05-12T11:57:20.343 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): ceph-grafana-dashboards-17.0.0-4087.g36c 112 kB/s | 24 kB 00:00 2021-05-12T11:57:20.500 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 211 kB/s | 34 kB 00:00 2021-05-12T11:57:20.652 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 638 kB/s | 196 kB 00:00 2021-05-12T11:57:20.696 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-mgr-dashboard-17.0.0-4087.g36c2cf2f 4.0 MB/s | 2.3 MB 00:00 2021-05-12T11:57:20.698 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:20.699 INFO:teuthology.orchestra.run.smithi090.stdout:Total 4.5 MB/s | 2.6 MB 00:00 2021-05-12T11:57:20.700 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:20.709 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:20.709 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:20.845 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:20.849 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:21.350 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-05-12T11:57:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-05-12T11:57:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-05-12T11:57:21.943 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.n 4/6 2021-05-12T11:57:22.961 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8. 5/6 2021-05-12T11:57:23.276 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 6/6 2021-05-12T11:57:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 6/6 2021-05-12T11:57:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8. 1/6 2021-05-12T11:57:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarc 2/6 2021-05-12T11:57:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.n 3/6 2021-05-12T11:57:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-05-12T11:57:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-05-12T11:57:23.758 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-05-12T11:57:23.758 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:23.958 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:23.958 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:23.958 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:23.959 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:23.959 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-05-12T11:57:23.959 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-05-12T11:57:23.959 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.4.1-12.el8.noarch 2021-05-12T11:57:23.960 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:23.960 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:24.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2021-05-12T11:57:24.581 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:25 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:25.316 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:25.317 INFO:teuthology.orchestra.run.smithi090.stdout:===================================================================================================== 2021-05-12T11:57:25.317 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:25.317 INFO:teuthology.orchestra.run.smithi090.stdout:===================================================================================================== 2021-05-12T11:57:25.318 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 7.4 M 2021-05-12T11:57:25.318 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:57:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-05-12T11:57:25.318 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-05-12T11:57:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-12T11:57:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-12T11:57:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-05-12T11:57:25.319 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-05-12T11:57:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-05-12T11:57:25.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:25.320 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:25.320 INFO:teuthology.orchestra.run.smithi090.stdout:===================================================================================================== 2021-05-12T11:57:25.320 INFO:teuthology.orchestra.run.smithi090.stdout:Install 8 Packages 2021-05-12T11:57:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:25.321 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 35 M 2021-05-12T11:57:25.322 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 194 M 2021-05-12T11:57:25.322 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:25.441 INFO:teuthology.orchestra.run.smithi090.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 35 MB/s | 4.3 MB 00:00 2021-05-12T11:57:25.470 INFO:teuthology.orchestra.run.smithi090.stdout:(2/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 29 MB/s | 4.3 MB 00:00 2021-05-12T11:57:25.477 INFO:teuthology.orchestra.run.smithi090.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 31 MB/s | 225 kB 00:00 2021-05-12T11:57:25.523 INFO:teuthology.orchestra.run.smithi090.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2021-05-12T11:57:25.539 INFO:teuthology.orchestra.run.smithi090.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 38 MB/s | 639 kB 00:00 2021-05-12T11:57:25.546 INFO:teuthology.orchestra.run.smithi090.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 28 MB/s | 169 kB 00:00 2021-05-12T11:57:25.747 INFO:teuthology.orchestra.run.smithi090.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 54 MB/s | 15 MB 00:00 2021-05-12T11:57:26.101 INFO:teuthology.orchestra.run.smithi090.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-408 9.4 MB/s | 7.4 MB 00:00 2021-05-12T11:57:26.104 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:26.105 INFO:teuthology.orchestra.run.smithi090.stdout:Total 45 MB/s | 35 MB 00:00 2021-05-12T11:57:26.113 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:26.132 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:26.132 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:26.446 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:26.450 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:27.589 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:27.750 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T11:57:27.888 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T11:57:28.052 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T11:57:28.554 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T11:57:28.741 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:57:29.248 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:57:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T11:57:30.130 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T11:57:30.431 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-05-12T11:57:30.633 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T11:57:32.634 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T11:57:33.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-05-12T11:57:34.082 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 8/8 2021-05-12T11:57:34.341 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 8/8 2021-05-12T11:57:34.341 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2 1/8 2021-05-12T11:57:34.341 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-05-12T11:57:34.341 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-05-12T11:57:34.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-05-12T11:57:34.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-05-12T11:57:34.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-05-12T11:57:34.342 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-05-12T11:57:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-05-12T11:57:34.614 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:34.816 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:34.816 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:34.817 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:34.817 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-05-12T11:57:34.817 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-05-12T11:57:34.817 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.3-5.el8.x86_64 2021-05-12T11:57:34.817 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-05-12T11:57:34.818 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-05-12T11:57:34.818 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-05-12T11:57:34.818 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-05-12T11:57:34.818 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:34.819 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:34.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2021-05-12T11:57:35.433 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:36 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:36.093 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:36.094 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2021-05-12T11:57:36.095 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:36.095 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2021-05-12T11:57:36.095 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:36.095 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 26 k 2021-05-12T11:57:36.095 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:57:36.096 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-05-12T11:57:36.096 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-05-12T11:57:36.096 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-05-12T11:57:36.096 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-05-12T11:57:36.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-05-12T11:57:36.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-05-12T11:57:36.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-05-12T11:57:36.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-05-12T11:57:36.097 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-05-12T11:57:36.098 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-05-12T11:57:36.098 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-05-12T11:57:36.098 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-05-12T11:57:36.098 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:36.099 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:36.099 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2021-05-12T11:57:36.099 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2021-05-12T11:57:36.099 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:36.100 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.9 M 2021-05-12T11:57:36.100 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2021-05-12T11:57:36.101 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:36.136 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 690 kB/s | 27 kB 00:00 2021-05-12T11:57:36.137 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 502 kB/s | 20 kB 00:00 2021-05-12T11:57:36.144 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 16 MB/s | 126 kB 00:00 2021-05-12T11:57:36.144 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 15 MB/s | 110 kB 00:00 2021-05-12T11:57:36.147 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 14 MB/s | 43 kB 00:00 2021-05-12T11:57:36.149 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 31 MB/s | 155 kB 00:00 2021-05-12T11:57:36.219 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): ceph-mgr-rook-17.0.0-4087.g36c2cf2f.el8 214 kB/s | 26 kB 00:00 2021-05-12T11:57:36.251 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 160 kB/s | 16 kB 00:00 2021-05-12T11:57:36.266 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 279 kB/s | 33 kB 00:00 2021-05-12T11:57:36.319 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.3 MB/s | 73 kB 00:00 2021-05-12T11:57:36.348 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-05-12T11:57:36.423 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-google-auth-1.1.1-10.el8.noarc 399 kB/s | 81 kB 00:00 2021-05-12T11:57:36.471 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.3 MB/s | 1.2 MB 00:00 2021-05-12T11:57:36.473 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:36.474 INFO:teuthology.orchestra.run.smithi090.stdout:Total 5.1 MB/s | 1.9 MB 00:00 2021-05-12T11:57:36.479 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:36.493 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:36.494 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:36.566 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:36.570 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:37.066 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:37.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-05-12T11:57:37.532 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-05-12T11:57:37.803 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-05-12T11:57:38.960 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-05-12T11:57:39.152 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-05-12T11:57:39.390 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-05-12T11:57:39.701 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-05-12T11:57:39.900 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-05-12T11:57:40.990 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-05-12T11:57:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-05-12T11:57:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-05-12T11:57:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-05-12T11:57:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 13/13 2021-05-12T11:57:42.139 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 13/13 2021-05-12T11:57:42.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/13 2021-05-12T11:57:42.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-05-12T11:57:42.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-05-12T11:57:42.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-05-12T11:57:42.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-05-12T11:57:42.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-05-12T11:57:42.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-05-12T11:57:42.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-05-12T11:57:42.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-05-12T11:57:42.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-05-12T11:57:42.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-05-12T11:57:42.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-05-12T11:57:42.458 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-05-12T11:57:42.459 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:42.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:42.666 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:42.666 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:42.666 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-05-12T11:57:42.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-05-12T11:57:42.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-05-12T11:57:42.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-05-12T11:57:42.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-05-12T11:57:42.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-05-12T11:57:42.668 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-05-12T11:57:42.668 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-05-12T11:57:42.668 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-05-12T11:57:42.668 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-05-12T11:57:42.669 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-05-12T11:57:42.669 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-05-12T11:57:42.669 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:42.669 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:42.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2021-05-12T11:57:43.307 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:44 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:43.969 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:43.970 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:43.971 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:43.971 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:43.971 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:43.972 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:17.0.0-4087.g36c2cf2f.el8 ceph-noarch 84 k 2021-05-12T11:57:43.972 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:57:43.972 INFO:teuthology.orchestra.run.smithi090.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-05-12T11:57:43.973 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-05-12T11:57:43.973 INFO:teuthology.orchestra.run.smithi090.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-05-12T11:57:43.973 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-05-12T11:57:43.973 INFO:teuthology.orchestra.run.smithi090.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-05-12T11:57:43.974 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:43.974 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2021-05-12T11:57:43.974 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:43.975 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.8 M 2021-05-12T11:57:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2021-05-12T11:57:43.976 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:44.112 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 3.8 MB/s | 538 kB 00:00 2021-05-12T11:57:44.225 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2021-05-12T11:57:44.226 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 187 kB/s | 21 kB 00:00 2021-05-12T11:57:44.226 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): ceph-mgr-cephadm-17.0.0-4087.g36c2cf2f.e 330 kB/s | 84 kB 00:00 2021-05-12T11:57:44.279 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 897 kB/s | 48 kB 00:00 2021-05-12T11:57:44.411 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2021-05-12T11:57:44.413 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:44.414 INFO:teuthology.orchestra.run.smithi090.stdout:Total 13 MB/s | 5.8 MB 00:00 2021-05-12T11:57:44.416 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:44.426 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:44.426 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:44.506 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:44.510 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:45.029 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-05-12T11:57:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-05-12T11:57:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-05-12T11:57:46.570 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-05-12T11:57:46.772 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-05-12T11:57:47.007 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 6/6 2021-05-12T11:57:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 6/6 2021-05-12T11:57:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 1/6 2021-05-12T11:57:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-05-12T11:57:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-05-12T11:57:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-05-12T11:57:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-05-12T11:57:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-05-12T11:57:47.514 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:47.740 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:47.740 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:47.740 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:17.0.0-4087.g36c2cf2f.el8.noarch 2021-05-12T11:57:47.741 INFO:teuthology.orchestra.run.smithi090.stdout: python3-apipkg-1.5-6.el8.noarch 2021-05-12T11:57:47.741 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.5.1-5.el8.noarch 2021-05-12T11:57:47.741 INFO:teuthology.orchestra.run.smithi090.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-05-12T11:57:47.741 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-05-12T11:57:47.742 INFO:teuthology.orchestra.run.smithi090.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-05-12T11:57:47.742 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:47.742 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:47.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2021-05-12T11:57:48.379 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:49 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:49.062 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:49.063 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:49.063 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2021-05-12T11:57:49.063 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:49.063 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:49.064 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 837 k 2021-05-12T11:57:49.064 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2021-05-12T11:57:49.064 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-05-12T11:57:49.064 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:49.064 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:49.065 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:49.065 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2021-05-12T11:57:49.065 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:49.065 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 919 k 2021-05-12T11:57:49.065 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 3.3 M 2021-05-12T11:57:49.066 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:49.116 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.5 MB/s | 82 kB 00:00 2021-05-12T11:57:49.300 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-17.0.0-4087.g36c2cf2f.el8.x86_ 3.5 MB/s | 837 kB 00:00 2021-05-12T11:57:49.301 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:49.302 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.8 MB/s | 919 kB 00:00 2021-05-12T11:57:49.303 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:49.324 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:49.325 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:49.399 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:49.402 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:49.916 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-05-12T11:57:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/2 2021-05-12T11:57:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2/2 2021-05-12T11:57:50.463 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/2 2021-05-12T11:57:50.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-05-12T11:57:50.722 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:50.946 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:50.947 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:50.947 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-05-12T11:57:50.947 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:50.947 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:51.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2021-05-12T11:57:51.687 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:52 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:52.351 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:52.352 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2021-05-12T11:57:52.352 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:52.353 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 125 k 2021-05-12T11:57:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:52.353 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:52.354 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:52.354 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:57:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:52.354 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 125 k 2021-05-12T11:57:52.354 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 439 k 2021-05-12T11:57:52.355 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:52.522 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-17.0.0-4087.g36c2cf2f.el8.x86_64 737 kB/s | 125 kB 00:00 2021-05-12T11:57:52.523 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:52.523 INFO:teuthology.orchestra.run.smithi090.stdout:Total 732 kB/s | 125 kB 00:00 2021-05-12T11:57:52.524 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:52.530 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:52.530 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:52.584 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:52.587 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:52.700 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:52.914 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:53.664 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:53.665 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:53.665 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:53.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:53.665 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:53.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2021-05-12T11:57:54.394 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:55 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:55.036 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:57:55.203 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:55.204 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:57:55.204 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:55.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2021-05-12T11:57:55.819 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:56 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:56.480 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:56.481 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:56.481 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2021-05-12T11:57:56.481 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:56.482 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:57:56.482 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 24 k 2021-05-12T11:57:56.482 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:56.482 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:57:56.483 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:57:56.483 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:57:56.483 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:56.483 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 24 k 2021-05-12T11:57:56.483 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 100 k 2021-05-12T11:57:56.484 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:57:56.603 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-17.0.0-4087.g36c2cf2f.el8.x86_6 195 kB/s | 24 kB 00:00 2021-05-12T11:57:56.603 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:57:56.604 INFO:teuthology.orchestra.run.smithi090.stdout:Total 193 kB/s | 24 kB 00:00 2021-05-12T11:57:56.604 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:57:56.608 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:57:56.608 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:57:56.610 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:57:56.613 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:57:56.661 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:57:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:57.322 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:57:57.322 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:57:57.531 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:57.531 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:57:57.531 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:57:57.532 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:57:57.532 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:57.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2021-05-12T11:57:58.158 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:59 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:58.669 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:57:58.837 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:57:58.838 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:57:58.838 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:57:58.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2021-05-12T11:57:59.466 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:00 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:57:59.952 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:58:00.121 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:00.122 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:58:00.122 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:00.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2021-05-12T11:58:00.733 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:01 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:01.630 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:58:01.798 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:01.799 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:58:01.800 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:02.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2021-05-12T11:58:02.600 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:03 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:03.086 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:58:03.255 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:03.256 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:58:03.256 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:03.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2021-05-12T11:58:03.873 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:04 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:04.369 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:58:04.538 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:04.539 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:58:04.540 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:04.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2021-05-12T11:58:05.156 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:06 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:05.702 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 is already installed. 2021-05-12T11:58:05.871 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:05.873 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2021-05-12T11:58:05.873 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:05.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2021-05-12T11:58:06.491 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:07 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:07.146 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:07.147 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:07.147 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2021-05-12T11:58:07.147 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:07.148 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:58:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 90 k 2021-05-12T11:58:07.148 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:07.148 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:58:07.148 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:07.149 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:58:07.149 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:07.149 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 90 k 2021-05-12T11:58:07.149 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 320 k 2021-05-12T11:58:07.149 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:58:07.306 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 568 kB/s | 90 kB 00:00 2021-05-12T11:58:07.307 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:58:07.307 INFO:teuthology.orchestra.run.smithi090.stdout:Total 564 kB/s | 90 kB 00:00 2021-05-12T11:58:07.308 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:58:07.319 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:58:07.320 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:58:07.369 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:58:07.373 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:58:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:58:08.205 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:08.445 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:08.703 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:08.704 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:58:08.943 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:08.944 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:58:08.944 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:58:08.944 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:08.945 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:09.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2021-05-12T11:58:09.674 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:10 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:10.386 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:10.387 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:10.387 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2021-05-12T11:58:10.388 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:10.388 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:58:10.388 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 3.7 M 2021-05-12T11:58:10.388 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:10.389 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:58:10.389 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:10.389 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:58:10.389 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:10.389 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.7 M 2021-05-12T11:58:10.390 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 21 M 2021-05-12T11:58:10.390 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:58:10.780 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 9.5 MB/s | 3.7 MB 00:00 2021-05-12T11:58:10.781 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:58:10.782 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.4 MB/s | 3.7 MB 00:00 2021-05-12T11:58:10.782 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:58:10.796 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:58:10.796 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:58:10.863 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:58:10.867 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:58:11.358 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:58:11.590 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:11.942 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:12.176 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:58:12.446 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:12.447 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:58:12.447 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:58:12.447 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:12.448 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:12.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2021-05-12T11:58:13.180 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:14 ago on Wed 12 May 2021 11:55:59 AM UTC. 2021-05-12T11:58:13.849 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2021-05-12T11:58:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2021-05-12T11:58:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:13.850 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2021-05-12T11:58:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:17.0.0-4087.g36c2cf2f.el8 ceph 171 k 2021-05-12T11:58:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2021-05-12T11:58:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2021-05-12T11:58:13.851 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2021-05-12T11:58:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 171 k 2021-05-12T11:58:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 671 k 2021-05-12T11:58:13.852 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2021-05-12T11:58:14.231 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-17.0.0-4087.g36c2cf2f.el8.x86_64.rpm 447 kB/s | 171 kB 00:00 2021-05-12T11:58:14.232 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2021-05-12T11:58:14.233 INFO:teuthology.orchestra.run.smithi090.stdout:Total 446 kB/s | 171 kB 00:00 2021-05-12T11:58:14.233 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2021-05-12T11:58:14.245 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2021-05-12T11:58:14.246 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2021-05-12T11:58:14.297 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2021-05-12T11:58:14.301 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2021-05-12T11:58:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2021-05-12T11:58:14.638 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:14.876 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 1/1 2021-05-12T11:58:15.133 INFO:teuthology.orchestra.run.smithi090.stdout:Installed products updated. 2021-05-12T11:58:15.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:15.375 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2021-05-12T11:58:15.375 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:17.0.0-4087.g36c2cf2f.el8.x86_64 2021-05-12T11:58:15.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-05-12T11:58:15.376 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2021-05-12T11:58:15.541 DEBUG:teuthology.parallel:result is None 2021-05-12T11:58:15.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2021-05-12T11:58:15.575 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS 2021-05-12T11:58:15.576 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T11:58:15.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=36c2cf2fb22a9790b2c097e158985773b35f1740 2021-05-12T11:58:15.724 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-05-12T11:58:15.743 INFO:teuthology.orchestra.run.smithi090.stdout:17.0.0-4087.g36c2cf2f.el8 2021-05-12T11:58:15.744 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-4087.g36c2cf2f.el8 2021-05-12T11:58:15.744 INFO:teuthology.task.install:The correct ceph version 17.0.0-4087.g36c2cf2f is installed. 2021-05-12T11:58:15.745 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-05-12T11:58:15.745 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:58:15.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-05-12T11:58:15.807 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-05-12T11:58:15.808 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:58:15.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2021-05-12T11:58:15.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-05-12T11:58:15.937 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-05-12T11:58:15.937 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:58:15.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2021-05-12T11:58:16.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-05-12T11:58:16.063 INFO:teuthology.run_tasks:Running task workunit... 2021-05-12T11:58:16.077 INFO:tasks.workunit:Pulling workunits from ref 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T11:58:16.078 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-05-12T11:58:16.078 INFO:tasks.workunit:timeout=3h 2021-05-12T11:58:16.078 INFO:tasks.workunit:cleanup=True 2021-05-12T11:58:16.079 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2021-05-12T11:58:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T11:58:16.117 INFO:teuthology.orchestra.run.smithi090.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-05-12T11:58:16.117 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-05-12T11:58:16.171 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-05-12T11:58:16.171 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-05-12T11:58:16.225 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T11:58:16.279 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-05-12T11:58:52.853 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '5cd0ad0001da189274219f5e829752717eb856d2'. 2021-05-12T11:58:52.853 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.853 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-05-12T11:58:52.854 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2021-05-12T11:58:52.854 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2021-05-12T11:58:52.854 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.854 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2021-05-12T11:58:52.854 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2021-05-12T11:58:52.855 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.855 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2021-05-12T11:58:52.855 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.855 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2021-05-12T11:58:52.855 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.856 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2021-05-12T11:58:52.856 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.856 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-05-12T11:58:52.856 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T11:58:52.856 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 5cd0ad0001d Merge branch 'negative_progress' of https://github.com/dvanders/ceph into wip-kefu-testing-2021-05-05-1449 2021-05-12T11:58:52.858 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-05-12T11:58:52.952 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2021-05-12T11:58:52.952 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-05-12T11:58:52.967 INFO:tasks.workunit:Running workunits matching osd on client.0... 2021-05-12T11:58:52.968 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2021-05-12T11:58:52.968 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2021-05-12T11:58:53.036 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T11:58:53.066 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T11:58:53.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:53.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:58:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T11:58:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T11:58:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T11:58:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T11:58:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T11:58:53.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T11:58:53.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/bad-inc-map 2021-05-12T11:58:53.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2021-05-12T11:58:53.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2021-05-12T11:58:53.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:58455 2021-05-12T11:58:53.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:58455 2021-05-12T11:58:53.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2021-05-12T11:58:53.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2021-05-12T11:58:53.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none ' 2021-05-12T11:58:53.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:58455 ' 2021-05-12T11:58:53.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2021-05-12T11:58:53.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2021-05-12T11:58:53.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T11:58:53.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2021-05-12T11:58:53.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2021-05-12T11:58:53.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2021-05-12T11:58:53.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2021-05-12T11:58:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2021-05-12T11:58:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:58:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:58:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:58:53.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:58:53.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:58:53.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:58:53.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:58:53.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:58:53.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:58:53.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:58:53.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:58:53.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:58:53.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:58:53.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:58:53.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:58:53.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:53.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:58:53.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:58:53.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:58:53.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:58:53.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:58:53.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:58:53.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:58:53.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:53.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:53.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24910 2021-05-12T11:58:53.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:58:53.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:58:53.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2021-05-12T11:58:53.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T11:58:53.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:53.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:53.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.24910 2021-05-12T11:58:53.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T11:58:53.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T11:58:53.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T11:58:53.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T11:58:53.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2021-05-12T11:58:53.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2021-05-12T11:58:53.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2021-05-12T11:58:53.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2021-05-12T11:58:53.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2021-05-12T11:58:53.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T11:58:53.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T11:58:53.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T11:58:53.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2021-05-12T11:58:53.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2021-05-12T11:58:53.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T11:58:53.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:53.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:53.456 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:53.457 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:53.457 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:53.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:58:53.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T11:58:53.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T11:58:53.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T11:58:53.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:53.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:53.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T11:58:53.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T11:58:53.676 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:53.676 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:53.677 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:53.677 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:53.677 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:53.677 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:53.677 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.24910/ceph-mon.a.asok 2021-05-12T11:58:53.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:53.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.24910/ceph-mon.a.asok config get fsid 2021-05-12T11:58:53.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T11:58:53.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:58:53.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:58:53.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T11:58:53.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T11:58:53.770 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:58:53.770 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:58:53.770 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:58:53.770 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:58:53.771 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:53.771 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:53.771 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.24910/ceph-mon.a.asok 2021-05-12T11:58:53.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:58:53.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.24910/ceph-mon.a.asok config get mon_host 2021-05-12T11:58:53.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2021-05-12T11:58:53.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2021-05-12T11:58:53.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T11:58:53.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T11:58:53.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T11:58:53.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2021-05-12T11:58:53.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T11:58:54.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T11:58:54.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:54.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:54.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:58:54.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:54.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:58:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2021-05-12T11:58:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:58:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:58:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T11:58:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:58:54.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2021-05-12T11:58:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 ' 2021-05-12T11:58:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:58:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:58:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:58:54.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2021-05-12T11:58:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2021-05-12T11:58:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:58:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:58:54.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:58:54.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:58:54.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:58:54.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:58:54.363 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:58:54.363 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:58:54.364 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:58:54.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:58:54.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:58:54.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:58:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:58:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:58:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:58:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:58:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:58:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:58:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:58:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:58:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/0 2021-05-12T11:58:54.367 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:58:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e757d0bf-e594-4863-85ca-25e2a6dae2dd 2021-05-12T11:58:54.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 e757d0bf-e594-4863-85ca-25e2a6dae2dd' 2021-05-12T11:58:54.368 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 e757d0bf-e594-4863-85ca-25e2a6dae2dd 2021-05-12T11:58:54.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:58:54.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD+wptgtJoEFhAAWfq+3teJ0t9T9xz8dBYYTg== 2021-05-12T11:58:54.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD+wptgtJoEFhAAWfq+3teJ0t9T9xz8dBYYTg=="}' 2021-05-12T11:58:54.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e757d0bf-e594-4863-85ca-25e2a6dae2dd -i td/bad-inc-map/0/new.json 2021-05-12T11:58:54.673 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:58:54.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/0/new.json 2021-05-12T11:58:54.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+wptgtJoEFhAAWfq+3teJ0t9T9xz8dBYYTg== --osd-uuid e757d0bf-e594-4863-85ca-25e2a6dae2dd 2021-05-12T11:58:54.705 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.703+0000 7f92f31800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:54.707 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.705+0000 7f92f31800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:54.709 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.707+0000 7f92f31800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:54.710 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.708+0000 7f92f31800c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:54.710 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.708+0000 7f92f31800c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:54.710 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.708+0000 7f92f31800c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T11:58:54.758 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:54.756+0000 7f92f31800c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2021-05-12T11:58:59.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/0/keyring 2021-05-12T11:58:59.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:58:59.339 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T11:58:59.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T11:58:59.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:58:59.580 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T11:58:59.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T11:58:59.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:58:59.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:58:59.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:58:59.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:58:59.600 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:59.598+0000 7f0d28efa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:59.620 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:59.618+0000 7f0d28efa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:59.622 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:58:59.619+0000 7f0d28efa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:58:59.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T11:58:59.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:58:59.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:58:59.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:58:59.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:58:59.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:58:59.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:58:59.818 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:58:59.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:58:59.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:00.141 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:00.139+0000 7f0d28efa0c0 -1 Falling back to public interface 2021-05-12T11:59:00.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:01.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:01.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:01.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:01.170 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:01.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:01.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:01.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:02.291 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:02.287+0000 7f0d28efa0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T11:59:02.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:02.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:02.530 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:02.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:02.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:02.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:02.939 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/26041,v1:127.0.0.1:6803/26041] [v2:127.0.0.1:6804/26041,v1:127.0.0.1:6805/26041] exists,up e757d0bf-e594-4863-85ca-25e2a6dae2dd 2021-05-12T11:59:02.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:02.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:02.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:02.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2021-05-12T11:59:02.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:59:02.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:02.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T11:59:02.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:02.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2021-05-12T11:59:02.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 ' 2021-05-12T11:59:02.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:02.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:02.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:02.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2021-05-12T11:59:02.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2021-05-12T11:59:02.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:02.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:02.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:59:02.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:02.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:02.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:02.948 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:02.949 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:02.949 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:59:02.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:59:02.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:59:02.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:02.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:02.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:02.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:59:02.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:59:02.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:02.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:02.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:02.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:02.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:02.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/1 2021-05-12T11:59:02.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:02.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=cb7d570e-b9cd-4731-a194-c2e3d36a49f3 2021-05-12T11:59:02.953 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 cb7d570e-b9cd-4731-a194-c2e3d36a49f3 2021-05-12T11:59:02.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 cb7d570e-b9cd-4731-a194-c2e3d36a49f3' 2021-05-12T11:59:02.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:02.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAGw5tgZB3BOBAA6wm9jZje7dlijwcLzDZ0cA== 2021-05-12T11:59:02.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAGw5tgZB3BOBAA6wm9jZje7dlijwcLzDZ0cA=="}' 2021-05-12T11:59:02.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new cb7d570e-b9cd-4731-a194-c2e3d36a49f3 -i td/bad-inc-map/1/new.json 2021-05-12T11:59:03.493 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:03.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/1/new.json 2021-05-12T11:59:03.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAGw5tgZB3BOBAA6wm9jZje7dlijwcLzDZ0cA== --osd-uuid cb7d570e-b9cd-4731-a194-c2e3d36a49f3 2021-05-12T11:59:03.526 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.523+0000 7fe0a7d2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:03.529 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.526+0000 7fe0a7d2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:03.534 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.528+0000 7fe0a7d2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:03.535 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.529+0000 7fe0a7d2f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:59:03.535 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.529+0000 7fe0a7d2f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:59:03.535 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.529+0000 7fe0a7d2f0c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T11:59:03.596 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:03.593+0000 7fe0a7d2f0c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2021-05-12T11:59:08.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/1/keyring 2021-05-12T11:59:08.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:08.109 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T11:59:08.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T11:59:08.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:08.605 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T11:59:08.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T11:59:08.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:08.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:08.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:08.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:08.625 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:08.622+0000 7fe90ac180c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.627 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:08.624+0000 7fe90ac180c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.628 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:08.625+0000 7fe90ac180c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:08.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T11:59:08.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:08.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:59:08.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:08.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:08.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:08.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:08.966 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:08.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:08.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:09.144 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:09.141+0000 7fe90ac180c0 -1 Falling back to public interface 2021-05-12T11:59:09.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:10.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:10.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:10.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:10.325 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:10.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:10.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:10.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:11.353 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:11.350+0000 7fe90ac180c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T11:59:11.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:11.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:11.705 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:11.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:11.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:11.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:12.059 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/27426,v1:127.0.0.1:6811/27426] [v2:127.0.0.1:6812/27426,v1:127.0.0.1:6813/27426] exists,up cb7d570e-b9cd-4731-a194-c2e3d36a49f3 2021-05-12T11:59:12.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:12.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:12.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:12.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2021-05-12T11:59:12.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T11:59:12.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:12.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T11:59:12.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:12.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2021-05-12T11:59:12.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 ' 2021-05-12T11:59:12.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2021-05-12T11:59:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2021-05-12T11:59:12.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:12.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:12.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T11:59:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:12.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:12.069 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:12.069 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:12.069 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:59:12.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:59:12.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' 2021-05-12T11:59:12.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:12.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:12.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:12.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T11:59:12.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T11:59:12.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/2 2021-05-12T11:59:12.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:12.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7785326b-b363-4f3d-ace8-4064e3b3512f 2021-05-12T11:59:12.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7785326b-b363-4f3d-ace8-4064e3b3512f' 2021-05-12T11:59:12.073 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 7785326b-b363-4f3d-ace8-4064e3b3512f 2021-05-12T11:59:12.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:12.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAQw5tgAGF2BBAAlE20rJF+4QOuCg5UT6whkg== 2021-05-12T11:59:12.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAQw5tgAGF2BBAAlE20rJF+4QOuCg5UT6whkg=="}' 2021-05-12T11:59:12.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7785326b-b363-4f3d-ace8-4064e3b3512f -i td/bad-inc-map/2/new.json 2021-05-12T11:59:12.636 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:12.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/2/new.json 2021-05-12T11:59:12.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQw5tgAGF2BBAAlE20rJF+4QOuCg5UT6whkg== --osd-uuid 7785326b-b363-4f3d-ace8-4064e3b3512f 2021-05-12T11:59:12.669 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.666+0000 7fde42f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.671 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.668+0000 7fde42f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.673 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.669+0000 7fde42f8d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:12.674 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.670+0000 7fde42f8d0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:59:12.674 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.670+0000 7fde42f8d0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:59:12.674 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.670+0000 7fde42f8d0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T11:59:12.721 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:12.719+0000 7fde42f8d0c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2021-05-12T11:59:17.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/2/keyring 2021-05-12T11:59:17.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:17.289 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T11:59:17.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T11:59:17.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:17.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T11:59:17.767 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T11:59:17.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=f5ecd35c-3d6a-4c1c-9827-62c09cc26c70 --auth-supported=none --mon-host=127.0.0.1:58455 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24910/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:17.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:17.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:17.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:17.786 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:17.783+0000 7fb8699550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:17.787 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:17.784+0000 7fb8699550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:17.789 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:17.786+0000 7fb8699550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:18.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T11:59:18.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:18.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:18.125 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:18.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:18.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:18.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:18.309 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:18.306+0000 7fb8699550c0 -1 Falling back to public interface 2021-05-12T11:59:18.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:19.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:19.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:19.481 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:19.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:19.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:19.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:19.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:20.509 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:20.506+0000 7fb8699550c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T11:59:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:20.841 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:20.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:20.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:20.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:21.219 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/28811,v1:127.0.0.1:6819/28811] [v2:127.0.0.1:6820/28811,v1:127.0.0.1:6821/28811] exists,up 7785326b-b363-4f3d-ace8-4064e3b3512f 2021-05-12T11:59:21.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:21.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:21.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:21.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2021-05-12T11:59:21.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2021-05-12T11:59:21.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2021-05-12T11:59:22.368 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'foo' already exists 2021-05-12T11:59:22.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T11:59:23.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2021-05-12T11:59:26.763 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 min_size to 1 2021-05-12T11:59:26.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2021-05-12T11:59:27.697 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 min_size to 2 2021-05-12T11:59:27.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2021-05-12T11:59:32.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:59:32.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2021-05-12T11:59:32.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:32.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:59:32.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:32.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:32.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:59:32.710 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:32.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:32.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:32.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:33.069 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/26041,v1:127.0.0.1:6803/26041] [v2:127.0.0.1:6804/26041,v1:127.0.0.1:6805/26041] exists,up e757d0bf-e594-4863-85ca-25e2a6dae2dd 2021-05-12T11:59:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:59:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2021-05-12T11:59:33.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:33.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:59:33.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:33.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:33.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:59:33.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:33.073 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:33.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:33.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:33.419 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/27426,v1:127.0.0.1:6811/27426] [v2:127.0.0.1:6812/27426,v1:127.0.0.1:6813/27426] exists,up cb7d570e-b9cd-4731-a194-c2e3d36a49f3 2021-05-12T11:59:33.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:33.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:33.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:33.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T11:59:33.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2021-05-12T11:59:33.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:33.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T11:59:33.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:33.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:33.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T11:59:33.424 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:33.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:33.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:33.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T11:59:33.788 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/28811,v1:127.0.0.1:6819/28811] [v2:127.0.0.1:6820/28811,v1:127.0.0.1:6821/28811] exists,up 7785326b-b363-4f3d-ace8-4064e3b3512f 2021-05-12T11:59:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2021-05-12T11:59:33.790 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:21.741+0000 7fb850860700 -1 osd.2 14 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.790 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:22.363+0000 7fb850860700 -1 osd.2 15 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.790 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:22.640+0000 7fb850860700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.791 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:23.368+0000 7fb850860700 -1 osd.2 18 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.791 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:23.370+0000 7fb850860700 -1 osd.2 18 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.791 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:25.515+0000 7fb84e85c700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.791 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:25.515+0000 7fb84e85c700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.791 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.432+0000 7fb84e85c700 -1 osd.2 20 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.792 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.904+0000 7fb84e85c700 -1 osd.2 21 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.792 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.904+0000 7fb84e85c700 -1 osd.2 21 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.792 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.179+0000 7fb84e85c700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.792 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.180+0000 7fb84e85c700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.793 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.205+0000 7fb84e85c700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.793 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.354+0000 7fb850860700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.793 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.691+0000 7fb850860700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.793 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.010+0000 7fb850860700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.793 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.011+0000 7fb850860700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.794 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.515+0000 7fb84e85c700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.794 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.516+0000 7fb84e85c700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-05-12T11:59:33.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2021-05-12T11:59:33.795 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:21.741+0000 7fb850860700 10 osd.2 14 handle_osd_map bailing because last < start (14<15) 2021-05-12T11:59:33.795 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:22.363+0000 7fb850860700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2021-05-12T11:59:33.795 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:22.640+0000 7fb850860700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-05-12T11:59:33.796 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:23.369+0000 7fb850860700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2021-05-12T11:59:33.796 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:23.370+0000 7fb850860700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2021-05-12T11:59:33.796 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:25.515+0000 7fb84e85c700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-05-12T11:59:33.796 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:25.515+0000 7fb84e85c700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-05-12T11:59:33.797 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.432+0000 7fb84e85c700 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2021-05-12T11:59:33.797 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.904+0000 7fb84e85c700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2021-05-12T11:59:33.797 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:26.905+0000 7fb84e85c700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2021-05-12T11:59:33.797 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.180+0000 7fb84e85c700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-05-12T11:59:33.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.180+0000 7fb84e85c700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-05-12T11:59:33.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.205+0000 7fb84e85c700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-05-12T11:59:33.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.354+0000 7fb850860700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-05-12T11:59:33.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:27.692+0000 7fb850860700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-05-12T11:59:33.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.010+0000 7fb850860700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-05-12T11:59:33.799 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.012+0000 7fb850860700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-05-12T11:59:33.799 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.516+0000 7fb84e85c700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-05-12T11:59:33.799 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T11:59:28.516+0000 7fb84e85c700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-05-12T11:59:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2021-05-12T11:59:33.800 INFO:tasks.workunit.client.0.smithi090.stdout:success 2021-05-12T11:59:33.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2021-05-12T11:59:33.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2021-05-12T11:59:33.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-05-12T11:59:34.281 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'foo' does not exist 2021-05-12T11:59:34.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2021-05-12T11:59:34.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:59:34.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:59:34.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:59:34.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:59:34.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:59:34.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:59:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2021-05-12T11:59:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:59:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:59:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:59:34.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:59:34.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:59:34.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:59:34.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:59:34.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:59:34.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:59:34.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:59:34.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:59:34.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:59:34.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:59:34.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:59:34.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:59:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:59:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:59:34.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:59:34.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:34.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:59:34.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24910 2021-05-12T11:59:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:59:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:59:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T11:59:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/bad-inc-map 0 2021-05-12T11:59:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T11:59:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T11:59:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T11:59:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:59:34.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:59:34.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:59:34.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:59:34.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:59:34.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:59:34.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:59:34.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:59:34.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:59:34.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:59:34.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:59:34.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:59:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:59:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:59:34.623 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:59:34.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T11:59:34.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T11:59:34.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:59:34.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:34.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24910 2021-05-12T11:59:34.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24910 2021-05-12T11:59:34.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:59:34.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:59:34.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T11:59:34.628 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T11:59:34.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T11:59:34.692 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2021-05-12T11:59:34.693 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2021-05-12T11:59:34.752 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T11:59:34.771 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T11:59:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:59:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T11:59:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T11:59:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T11:59:34.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T11:59:34.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T11:59:34.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T11:59:34.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T11:59:34.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/divergent-priors 2021-05-12T11:59:34.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2021-05-12T11:59:34.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2021-05-12T11:59:34.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2021-05-12T11:59:34.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2021-05-12T11:59:34.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2021-05-12T11:59:34.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2021-05-12T11:59:34.778 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2021-05-12T11:59:34.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2021-05-12T11:59:34.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2021-05-12T11:59:34.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2021-05-12T11:59:34.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2021-05-12T11:59:34.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2021-05-12T11:59:34.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2021-05-12T11:59:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2021-05-12T11:59:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2021-05-12T11:59:34.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2021-05-12T11:59:34.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2021-05-12T11:59:34.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2021-05-12T11:59:34.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2021-05-12T11:59:34.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2021-05-12T11:59:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2021-05-12T11:59:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2021-05-12T11:59:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2021-05-12T11:59:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2021-05-12T11:59:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2021-05-12T11:59:34.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2021-05-12T11:59:34.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none ' 2021-05-12T11:59:34.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-05-12T11:59:34.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-05-12T11:59:34.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2021-05-12T11:59:34.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:34.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2021-05-12T11:59:34.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T11:59:34.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2021-05-12T11:59:34.789 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_divergent_2 2021-05-12T11:59:34.790 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_divergent_3 2021-05-12T11:59:34.790 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_divergent_ec' 2021-05-12T11:59:34.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T11:59:34.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T11:59:34.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T11:59:34.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T11:59:34.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T11:59:34.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T11:59:34.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T11:59:34.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T11:59:34.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T11:59:34.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T11:59:34.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T11:59:34.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T11:59:34.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T11:59:34.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T11:59:34.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T11:59:34.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T11:59:34.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T11:59:34.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T11:59:34.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T11:59:34.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T11:59:34.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T11:59:34.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T11:59:34.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T11:59:34.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T11:59:34.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T11:59:34.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T11:59:34.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:34.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:34.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T11:59:34.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T11:59:34.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T11:59:34.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T11:59:34.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T11:59:34.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:34.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:34.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29659 2021-05-12T11:59:34.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T11:59:34.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T11:59:34.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T11:59:34.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T11:59:34.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T11:59:34.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2021-05-12T11:59:34.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2021-05-12T11:59:34.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2021-05-12T11:59:34.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2021-05-12T11:59:34.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2021-05-12T11:59:34.818 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2021-05-12T11:59:34.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2021-05-12T11:59:34.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2021-05-12T11:59:34.819 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T11:59:34.820 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T11:59:34.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2021-05-12T11:59:34.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T11:59:34.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T11:59:34.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T11:59:34.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T11:59:34.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T11:59:34.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T11:59:35.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T11:59:35.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:35.154 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:35.154 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:35.154 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:35.154 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:35.155 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:35.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T11:59:35.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T11:59:35.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T11:59:35.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:59:35.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:59:35.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T11:59:35.420 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T11:59:35.420 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:59:35.420 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:59:35.421 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:59:35.421 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:35.422 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:35.422 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:35.422 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T11:59:35.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:59:35.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get fsid 2021-05-12T11:59:35.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T11:59:35.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T11:59:35.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T11:59:35.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T11:59:35.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T11:59:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T11:59:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T11:59:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T11:59:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T11:59:35.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:35.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:35.512 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T11:59:35.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T11:59:35.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get mon_host 2021-05-12T11:59:35.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2021-05-12T11:59:35.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T11:59:35.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T11:59:35.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T11:59:35.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T11:59:35.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T11:59:35.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T11:59:36.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T11:59:36.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:36.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:36.476 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:36.476 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:36.477 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:36.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:36.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:59:36.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T11:59:36.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:59:36.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2021-05-12T11:59:36.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:59:36.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:36.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T11:59:36.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:36.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T11:59:36.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:36.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:36.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:36.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:36.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T11:59:36.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T11:59:36.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:36.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:36.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:59:36.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:36.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:36.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:36.518 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:36.519 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:36.519 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:36.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:36.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:36.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:36.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:36.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:36.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:59:36.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:59:36.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:36.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:36.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:36.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:36.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:36.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T11:59:36.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:36.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T11:59:36.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 ae18c233-dcbd-4a25-9ded-fc69c9202457' 2021-05-12T11:59:36.525 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T11:59:36.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:36.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAow5tgvis/HxAA28BseHDFDgU+8dbaLkEDBA== 2021-05-12T11:59:36.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAow5tgvis/HxAA28BseHDFDgU+8dbaLkEDBA=="}' 2021-05-12T11:59:36.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ae18c233-dcbd-4a25-9ded-fc69c9202457 -i td/divergent-priors/0/new.json 2021-05-12T11:59:36.821 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:36.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T11:59:36.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAow5tgvis/HxAA28BseHDFDgU+8dbaLkEDBA== --osd-uuid ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T11:59:36.862 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.860+0000 7f4d514af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:36.864 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.862+0000 7f4d514af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:36.865 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.863+0000 7f4d514af0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:36.866 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.864+0000 7f4d514af0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:59:36.866 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.864+0000 7f4d514af0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:59:36.866 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.864+0000 7f4d514af0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T11:59:36.906 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:36.903+0000 7f4d514af0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T11:59:41.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T11:59:41.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:41.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T11:59:41.221 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T11:59:41.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:41.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T11:59:41.700 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T11:59:41.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:41.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:41.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:41.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:41.727 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:41.725+0000 7fc179ece0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:41.739 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:41.737+0000 7fc179ece0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:41.740 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:41.738+0000 7fc179ece0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:42.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T11:59:42.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:42.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T11:59:42.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:42.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:42.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:42.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:42.072 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:42.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:42.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:42.256 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:42.253+0000 7fc179ece0c0 -1 Falling back to public interface 2021-05-12T11:59:42.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:43.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:43.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:43.448 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:43.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:43.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:43.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:43.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:44.469 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:44.466+0000 7fc179ece0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T11:59:44.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:44.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:44.821 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:44.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:44.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:44.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T11:59:45.183 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30786,v1:127.0.0.1:6803/30786] [v2:127.0.0.1:6804/30786,v1:127.0.0.1:6805/30786] exists,up ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T11:59:45.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:45.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:45.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:45.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:59:45.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2021-05-12T11:59:45.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:59:45.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:45.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T11:59:45.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:45.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T11:59:45.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:45.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:45.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:45.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:45.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T11:59:45.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T11:59:45.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:45.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:45.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:59:45.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:45.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:45.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:45.193 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:45.193 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:45.194 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:45.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:45.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:45.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:45.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:45.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:45.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:59:45.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:59:45.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:45.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:45.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:45.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:45.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:45.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T11:59:45.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:45.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T11:59:45.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 1ccf09c6-fc95-4724-bf58-2a61dee625e6' 2021-05-12T11:59:45.198 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T11:59:45.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:45.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAxw5tgVFYfDBAAnFWQ2WNvs1IyTTygOfhUkg== 2021-05-12T11:59:45.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAxw5tgVFYfDBAAnFWQ2WNvs1IyTTygOfhUkg=="}' 2021-05-12T11:59:45.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1ccf09c6-fc95-4724-bf58-2a61dee625e6 -i td/divergent-priors/1/new.json 2021-05-12T11:59:45.677 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:45.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T11:59:45.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAxw5tgVFYfDBAAnFWQ2WNvs1IyTTygOfhUkg== --osd-uuid 1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T11:59:45.719 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.717+0000 7f934d4f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:45.721 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.719+0000 7f934d4f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:45.722 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.720+0000 7f934d4f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:45.722 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.720+0000 7f934d4f80c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:45.723 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.720+0000 7f934d4f80c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:45.723 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.720+0000 7f934d4f80c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T11:59:45.764 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:45.762+0000 7f934d4f80c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T11:59:50.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T11:59:50.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:50.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T11:59:50.550 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T11:59:50.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T11:59:51.039 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T11:59:51.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T11:59:51.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:51.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T11:59:51.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T11:59:51.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T11:59:51.067 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:51.065+0000 7f851c5fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:51.086 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:51.084+0000 7f851c5fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:51.087 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:51.085+0000 7f851c5fe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:51.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T11:59:51.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T11:59:51.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T11:59:51.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T11:59:51.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T11:59:51.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:51.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T11:59:51.407 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T11:59:51.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:51.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:51.603 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:51.601+0000 7f851c5fe0c0 -1 Falling back to public interface 2021-05-12T11:59:51.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:52.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:52.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:52.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T11:59:52.783 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T11:59:52.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:52.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:53.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T11:59:53.826 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:53.823+0000 7f851c5fe0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T11:59:54.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T11:59:54.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T11:59:54.167 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:54.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T11:59:54.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T11:59:54.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T11:59:54.556 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32171,v1:127.0.0.1:6811/32171] [v2:127.0.0.1:6812/32171,v1:127.0.0.1:6813/32171] exists,up 1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T11:59:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T11:59:54.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T11:59:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T11:59:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T11:59:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2021-05-12T11:59:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T11:59:54.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T11:59:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T11:59:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T11:59:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T11:59:54.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T11:59:54.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T11:59:54.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T11:59:54.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T11:59:54.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T11:59:54.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T11:59:54.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T11:59:54.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T11:59:54.565 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T11:59:54.565 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T11:59:54.565 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T11:59:54.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T11:59:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T11:59:54.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T11:59:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T11:59:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T11:59:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T11:59:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T11:59:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T11:59:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T11:59:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T11:59:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T11:59:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T11:59:54.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T11:59:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T11:59:54.569 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T11:59:54.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7537394c-b095-4135-98fc-d1a8fc5a0322' 2021-05-12T11:59:54.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T11:59:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA6w5tgqrx0IhAAejqMJ6TlXN6WDjB0JZecQw== 2021-05-12T11:59:54.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA6w5tgqrx0IhAAejqMJ6TlXN6WDjB0JZecQw=="}' 2021-05-12T11:59:54.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7537394c-b095-4135-98fc-d1a8fc5a0322 -i td/divergent-priors/2/new.json 2021-05-12T11:59:55.138 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T11:59:55.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T11:59:55.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA6w5tgqrx0IhAAejqMJ6TlXN6WDjB0JZecQw== --osd-uuid 7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T11:59:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.177+0000 7fde7c5ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:55.187 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.185+0000 7fde7c5ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:55.188 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.186+0000 7fde7c5ea0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T11:59:55.189 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.186+0000 7fde7c5ea0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:55.189 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.186+0000 7fde7c5ea0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:55.189 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.186+0000 7fde7c5ea0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T11:59:55.254 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T11:59:55.252+0000 7fde7c5ea0c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T11:59:59.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T11:59:59.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T11:59:59.802 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T11:59:59.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T11:59:59.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:00:00.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:00:00.291 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:00:00.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:00.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:00:00.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:00:00.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:00:00.318 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:00.316+0000 7fc0adc4a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:00.320 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:00.318+0000 7fc0adc4a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:00.322 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:00.319+0000 7fc0adc4a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:00.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:00:00.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:00:00.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:00:00.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:00:00.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:00:00.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:00.665 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:00:00.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:00:00.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:00.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:00.839 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:00.837+0000 7fc0adc4a0c0 -1 Falling back to public interface 2021-05-12T12:00:01.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:02.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:02.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:02.040 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:00:02.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:00:02.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:02.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:02.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:03.066 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:03.063+0000 7fc0adc4a0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:00:03.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:03.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:03.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:00:03.408 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:00:03.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:03.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:03.830 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/33564,v1:127.0.0.1:6819/33564] [v2:127.0.0.1:6820/33564,v1:127.0.0.1:6821/33564] exists,up 7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T12:00:03.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:00:03.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:00:03.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:00:03.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2021-05-12T12:00:04.498 INFO:tasks.workunit.client.0.smithi090.stderr:noout is set 2021-05-12T12:00:04.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2021-05-12T12:00:05.251 INFO:tasks.workunit.client.0.smithi090.stderr:noin is set 2021-05-12T12:00:05.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2021-05-12T12:00:05.936 INFO:tasks.workunit.client.0.smithi090.stderr:nodown is set 2021-05-12T12:00:05.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2021-05-12T12:00:05.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T12:00:06.537 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' already exists 2021-05-12T12:00:06.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:00:07.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2021-05-12T12:00:08.333 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 3 2021-05-12T12:00:08.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2021-05-12T12:00:09.033 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 min_size to 2 2021-05-12T12:00:09.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2021-05-12T12:00:09.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:00:09.043 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:00:09.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:00:09.408 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:00:09.408 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:00:09.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:00:09.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:09.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:00:09.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T12:00:09.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T12:00:09.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T12:00:09.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:09.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:00:09.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705670 2021-05-12T12:00:09.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705670 2021-05-12T12:00:09.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-38654705670' 2021-05-12T12:00:09.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:09.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:00:09.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542148 2021-05-12T12:00:09.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542148 2021-05-12T12:00:09.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-38654705670 2-60129542148' 2021-05-12T12:00:09.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:09.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T12:00:09.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:09.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:00:09.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T12:00:09.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:09.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T12:00:09.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T12:00:09.814 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836488 2021-05-12T12:00:09.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:10.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836486 -lt 21474836488 2021-05-12T12:00:10.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:00:11.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:00:11.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:11.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836489 -lt 21474836488 2021-05-12T12:00:11.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:11.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705670 2021-05-12T12:00:11.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:11.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:00:11.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705670 2021-05-12T12:00:11.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:11.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705670 2021-05-12T12:00:11.600 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705670 2021-05-12T12:00:11.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2021-05-12T12:00:11.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:00:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705671 -lt 38654705670 2021-05-12T12:00:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2021-05-12T12:00:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:11.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:00:11.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542148 2021-05-12T12:00:11.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:11.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542148 2021-05-12T12:00:11.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2021-05-12T12:00:11.970 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542148 2021-05-12T12:00:11.970 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:00:12.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542148 -lt 60129542148 2021-05-12T12:00:12.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2021-05-12T12:00:12.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:00:12.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:00:12.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:00:12.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:00:12.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:00:12.336 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:00:12.337 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:00:12.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:00:12.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:00:12.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:00:12.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:00:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:00:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:00:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:00:12.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:00:12.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:00:12.785 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:00:12.785 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:00:12.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:00:12.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:12.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:00:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T12:00:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T12:00:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T12:00:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:12.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:00:13.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705673 2021-05-12T12:00:13.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705673 2021-05-12T12:00:13.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-38654705673' 2021-05-12T12:00:13.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:13.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:00:13.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542150 2021-05-12T12:00:13.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542150 2021-05-12T12:00:13.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-38654705673 2-60129542150' 2021-05-12T12:00:13.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:13.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T12:00:13.218 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:13.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:00:13.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T12:00:13.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:13.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T12:00:13.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T12:00:13.220 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836491 2021-05-12T12:00:13.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:13.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T12:00:13.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:13.590 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705673 2021-05-12T12:00:13.590 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:13.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:00:13.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705673 2021-05-12T12:00:13.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:13.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705673 2021-05-12T12:00:13.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2021-05-12T12:00:13.593 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705673 2021-05-12T12:00:13.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:00:13.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705671 -lt 38654705673 2021-05-12T12:00:13.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:00:14.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:00:14.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:00:15.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705673 -lt 38654705673 2021-05-12T12:00:15.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:15.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542150 2021-05-12T12:00:15.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:15.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:00:15.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542150 2021-05-12T12:00:15.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:15.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542150 2021-05-12T12:00:15.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2021-05-12T12:00:15.342 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542150 2021-05-12T12:00:15.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:00:15.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542151 -lt 60129542150 2021-05-12T12:00:15.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:00:15.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:15.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:16.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:00:16.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:16.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:16.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:16.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:16.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:16.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:16.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:16.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:00:16.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:16.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:16.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:17.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:00:17.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:00:17.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:00:17.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2021-05-12T12:00:17.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2021-05-12T12:00:17.375 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:00:17.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2021-05-12T12:00:17.384 INFO:tasks.workunit.client.0.smithi090.stdout:primary and soon to be divergent is 1 2021-05-12T12:00:17.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2021-05-12T12:00:17.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:17.723 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:17.723 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:00:07.197700+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:00:17.723 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:00:17.723 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:17.724 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2021-05-12T12:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T12:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2021-05-12T12:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2021-05-12T12:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T12:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2021-05-12T12:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2021-05-12T12:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T12:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2021-05-12T12:00:17.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2021-05-12T12:00:17.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2021-05-12T12:00:17.737 INFO:tasks.workunit.client.0.smithi090.stdout:writing initial objects 2021-05-12T12:00:17.738 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2021-05-12T12:00:17.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:17.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2021-05-12T12:00:17.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:17.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2021-05-12T12:00:17.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:17.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2021-05-12T12:00:18.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2021-05-12T12:00:18.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2021-05-12T12:00:18.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2021-05-12T12:00:18.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2021-05-12T12:00:18.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2021-05-12T12:00:18.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2021-05-12T12:00:18.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2021-05-12T12:00:18.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:18.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2021-05-12T12:00:19.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2021-05-12T12:00:19.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2021-05-12T12:00:19.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2021-05-12T12:00:19.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2021-05-12T12:00:19.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2021-05-12T12:00:19.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2021-05-12T12:00:19.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2021-05-12T12:00:19.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:19.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2021-05-12T12:00:20.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2021-05-12T12:00:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2021-05-12T12:00:20.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2021-05-12T12:00:20.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2021-05-12T12:00:20.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2021-05-12T12:00:20.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2021-05-12T12:00:20.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2021-05-12T12:00:20.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:20.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2021-05-12T12:00:21.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2021-05-12T12:00:21.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2021-05-12T12:00:21.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2021-05-12T12:00:21.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2021-05-12T12:00:21.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2021-05-12T12:00:21.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2021-05-12T12:00:21.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2021-05-12T12:00:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2021-05-12T12:00:22.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2021-05-12T12:00:22.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2021-05-12T12:00:22.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2021-05-12T12:00:22.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2021-05-12T12:00:22.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2021-05-12T12:00:22.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2021-05-12T12:00:22.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2021-05-12T12:00:22.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:22.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2021-05-12T12:00:23.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2021-05-12T12:00:23.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2021-05-12T12:00:23.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2021-05-12T12:00:23.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2021-05-12T12:00:23.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2021-05-12T12:00:23.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2021-05-12T12:00:23.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2021-05-12T12:00:23.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2021-05-12T12:00:23.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:23.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2021-05-12T12:00:24.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2021-05-12T12:00:24.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2021-05-12T12:00:24.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2021-05-12T12:00:24.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2021-05-12T12:00:24.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2021-05-12T12:00:24.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2021-05-12T12:00:24.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2021-05-12T12:00:24.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:24.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2021-05-12T12:00:25.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2021-05-12T12:00:25.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2021-05-12T12:00:25.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2021-05-12T12:00:25.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2021-05-12T12:00:25.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2021-05-12T12:00:25.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2021-05-12T12:00:25.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2021-05-12T12:00:25.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:25.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2021-05-12T12:00:26.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2021-05-12T12:00:26.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2021-05-12T12:00:26.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2021-05-12T12:00:26.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2021-05-12T12:00:26.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2021-05-12T12:00:26.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2021-05-12T12:00:26.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2021-05-12T12:00:26.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2021-05-12T12:00:26.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:26.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2021-05-12T12:00:27.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2021-05-12T12:00:27.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2021-05-12T12:00:27.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2021-05-12T12:00:27.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2021-05-12T12:00:27.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2021-05-12T12:00:27.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2021-05-12T12:00:27.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2021-05-12T12:00:27.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:27.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2021-05-12T12:00:28.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2021-05-12T12:00:28.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2021-05-12T12:00:28.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2021-05-12T12:00:28.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2021-05-12T12:00:28.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2021-05-12T12:00:28.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2021-05-12T12:00:28.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2021-05-12T12:00:28.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:28.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2021-05-12T12:00:29.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2021-05-12T12:00:29.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2021-05-12T12:00:29.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2021-05-12T12:00:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2021-05-12T12:00:29.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2021-05-12T12:00:29.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2021-05-12T12:00:29.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T12:00:29.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T12:00:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:00:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2021-05-12T12:00:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:00:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:00:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:00:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:00:29.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:00:29.884 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:00:29.884 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:00:29.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:00:29.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:00:29.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:00:29.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:00:29.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:00:29.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:00:29.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:00:29.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:00:30.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:00:30.298 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:00:30.298 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:00:30.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:00:30.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:30.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:00:30.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T12:00:30.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T12:00:30.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T12:00:30.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:30.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:00:30.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705679 2021-05-12T12:00:30.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705679 2021-05-12T12:00:30.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-38654705679' 2021-05-12T12:00:30.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:30.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:00:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542156 2021-05-12T12:00:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542156 2021-05-12T12:00:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-38654705679 2-60129542156' 2021-05-12T12:00:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:30.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T12:00:30.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:00:30.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T12:00:30.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:30.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T12:00:30.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T12:00:30.732 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836496 2021-05-12T12:00:30.732 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:31.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836497 -lt 21474836496 2021-05-12T12:00:31.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:31.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705679 2021-05-12T12:00:31.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:31.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:00:31.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705679 2021-05-12T12:00:31.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:31.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705679 2021-05-12T12:00:31.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2021-05-12T12:00:31.100 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705679 2021-05-12T12:00:31.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:00:31.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705679 -lt 38654705679 2021-05-12T12:00:31.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:31.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542156 2021-05-12T12:00:31.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:31.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:00:31.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542156 2021-05-12T12:00:31.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:31.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542156 2021-05-12T12:00:31.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2021-05-12T12:00:31.483 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542156 2021-05-12T12:00:31.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:00:31.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542156 -lt 60129542156 2021-05-12T12:00:31.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:00:31.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:31.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:32.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:00:32.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:00:32.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:00:32.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:00:32.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:00:32.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:00:32.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:00:32.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:00:32.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:00:32.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:00:32.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:00:32.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:00:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:00:33.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:00:33.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:00:33.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2021-05-12T12:00:33.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:00:33.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:00:33.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:00:33.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:00:33.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2021-05-12T12:00:33.558 INFO:tasks.workunit.client.0.smithi090.stdout:blackholing osds 0 2 2021-05-12T12:00:33.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2021-05-12T12:00:33.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:33.920 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:33.921 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:00:07.197700+0000 27'100 27:113 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:00:33.921 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:00:33.921 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:33.922 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:00:33.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:33.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2021-05-12T12:00:33.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:00:33.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:00:33.933 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:33.933 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:33.934 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:33.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:00:33.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T12:00:33.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:00:34.024 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:00:34.024 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:00:34.024 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:00:34.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:34.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2021-05-12T12:00:34.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:00:34.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:00:34.034 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:34.034 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:34.034 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:34.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:00:34.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T12:00:34.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:00:34.124 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:00:34.124 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:00:34.125 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:00:34.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2021-05-12T12:00:34.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2021-05-12T12:00:34.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2021-05-12T12:00:34.134 INFO:tasks.workunit.client.0.smithi090.stdout:writing divergent object 2021-05-12T12:00:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2021-05-12T12:00:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2021-05-12T12:00:34.136 INFO:tasks.workunit.client.0.smithi090.stdout:create missing divergent object 2021-05-12T12:00:34.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T12:00:34.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2021-05-12T12:00:34.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local pooltype=rep 2021-05-12T12:00:34.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2021-05-12T12:00:34.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local which=data 2021-05-12T12:00:34.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2021-05-12T12:00:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local poolname=test 2021-05-12T12:00:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2021-05-12T12:00:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: local objname=existing_99 2021-05-12T12:00:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: shift 2021-05-12T12:00:34.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local dir=td/divergent-priors 2021-05-12T12:00:34.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: shift 2021-05-12T12:00:34.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: local shard_id=0 2021-05-12T12:00:34.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: shift 2021-05-12T12:00:34.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-05-12T12:00:34.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: get_osds test existing_99 2021-05-12T12:00:34.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local poolname=test 2021-05-12T12:00:34.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: local objectname=existing_99 2021-05-12T12:00:34.142 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: ceph --format json osd map test existing_99 2021-05-12T12:00:34.143 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: jq '.acting | .[]' 2021-05-12T12:00:34.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local 'osds=1 2021-05-12T12:00:34.505 INFO:tasks.workunit.client.0.smithi090.stderr:0 2021-05-12T12:00:34.506 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:00:34.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: get_osds: echo 1 0 2 2021-05-12T12:00:34.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local -a initial_osds 2021-05-12T12:00:34.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local osd_id=1 2021-05-12T12:00:34.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: '[' rep '!=' ec ']' 2021-05-12T12:00:34.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: shard_id= 2021-05-12T12:00:34.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: cat td/divergent-priors/1/type 2021-05-12T12:00:34.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: type=bluestore 2021-05-12T12:00:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-05-12T12:00:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local daemon=osd 2021-05-12T12:00:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local id=1 2021-05-12T12:00:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local config=bluestore_debug_inject_read_err 2021-05-12T12:00:34.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local value=true 2021-05-12T12:00:34.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: jq 'has("success")' 2021-05-12T12:00:34.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: get_asok_path osd.1 2021-05-12T12:00:34.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:00:34.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:00:34.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:34.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:34.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:34.512 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:00:34.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.29659/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-05-12T12:00:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: test true == true 2021-05-12T12:00:34.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: local loop=0 2021-05-12T12:00:34.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: grep -q Invalid 2021-05-12T12:00:34.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: get_asok_path osd.1 2021-05-12T12:00:34.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:00:34.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:00:34.611 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:00:34.611 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:34.611 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:34.612 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:00:34.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: CEPH_ARGS= 2021-05-12T12:00:34.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok injectdataerr test existing_99 2021-05-12T12:00:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2021-05-12T12:00:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2021-05-12T12:00:44.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2021-05-12T12:00:44.752 INFO:tasks.workunit.client.0.smithi090.stderr:rados: no process found 2021-05-12T12:00:44.752 INFO:tasks.workunit.client.0.smithi090.stdout:killing all the osds 2021-05-12T12:00:44.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2021-05-12T12:00:44.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:45.105 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:45.106 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2021-05-12T12:00:38.310749+0000 27'101 27:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:00:45.106 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:00:45.106 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:45.107 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:00:45.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:00:45.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:00:45.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:00:45.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:00:45.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:00:45.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:00:45.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:00:45.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:45.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2021-05-12T12:00:45.742 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:00:45.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:45.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2021-05-12T12:00:46.255 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:00:46.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T12:00:46.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2021-05-12T12:00:46.761 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:00:46.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:46.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2021-05-12T12:00:47.369 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already out. 2021-05-12T12:00:47.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:47.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2021-05-12T12:00:47.921 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already out. 2021-05-12T12:00:47.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2021-05-12T12:00:47.931 INFO:tasks.workunit.client.0.smithi090.stdout:bringing up non_divergent 0 2 2021-05-12T12:00:47.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2021-05-12T12:00:48.272 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:00:48.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2021-05-12T12:00:38.310749+0000 27'101 27:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:00:48.273 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:00:48.273 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:00:48.274 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:00:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2021-05-12T12:00:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:00:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:00:48.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:00:48.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:00:48.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:00:48.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:00:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:00:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:00:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:00:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:00:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:00:48.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:00:48.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:00:48.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:00:48.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:00:48.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:00:48.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:00:48.291 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:00:48.291 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:48.292 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:48.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:00:48.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:00:48.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:00:48.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:00:48.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:00:48.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:00:48.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:00:48.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:00:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:00:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:00:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:00:48.295 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:00:48.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:48.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:00:48.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:00:48.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:00:48.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:00:48.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:00:48.314 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:48.312+0000 7f6f063d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:48.315 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:48.313+0000 7f6f063d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:48.317 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:48.315+0000 7f6f063d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:48.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:00:48.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:00:48.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:00:48.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:00:48.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:00:48.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:48.653 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:00:48.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:00:48.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:48.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:48.834 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:48.831+0000 7f6f063d10c0 -1 Falling back to public interface 2021-05-12T12:00:49.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:50.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:50.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:50.040 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:00:50.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:00:50.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:50.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:50.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:51.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:51.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:51.411 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:00:51.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:00:51.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:51.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:51.618 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:51.616+0000 7f6f063d10c0 -1 osd.0 27 log_to_monitors {default=true} 2021-05-12T12:00:51.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:52.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:52.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:52.783 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:00:52.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:00:52.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:52.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:00:53.153 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/37583,v1:127.0.0.1:6803/37583] [v2:127.0.0.1:6804/37583,v1:127.0.0.1:6805/37583] exists,up ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T12:00:53.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:00:53.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:00:53.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:00:53.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:53.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2021-05-12T12:00:53.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:00:53.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:00:53.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:00:53.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:00:53.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:00:53.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:00:53.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:00:53.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:00:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:00:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:00:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:00:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:00:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:00:53.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:00:53.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:00:53.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:00:53.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:00:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:00:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:00:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:00:53.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:00:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:00:53.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:00:53.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:00:53.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:00:53.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:00:53.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:00:53.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:53.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:00:53.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:00:53.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:00:53.165 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:00:53.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:00:53.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:00:53.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:00:53.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:00:53.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:00:53.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:00:53.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:00:53.183 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:53.181+0000 7fef4a9360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:53.185 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:53.183+0000 7fef4a9360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:53.186 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:53.184+0000 7fef4a9360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:00:53.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:00:53.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:00:53.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:00:53.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:00:53.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:00:53.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:53.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:00:53.542 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:00:53.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:53.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:53.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:53.699+0000 7fef4a9360c0 -1 Falling back to public interface 2021-05-12T12:00:53.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:54.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:54.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:54.916 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:00:54.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:00:54.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:54.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:55.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:00:56.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:00:56.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:00:56.288 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:00:56.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:00:56.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:00:56.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:00:56.362 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:00:56.360+0000 7fef4a9360c0 -1 osd.2 27 log_to_monitors {default=true} 2021-05-12T12:00:56.663 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/38166,v1:127.0.0.1:6811/38166] [v2:127.0.0.1:6812/38166,v1:127.0.0.1:6813/38166] exists,up 7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T12:00:56.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:00:56.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:00:56.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:00:56.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:56.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2021-05-12T12:00:57.160 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already in. 2021-05-12T12:00:57.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T12:00:57.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2021-05-12T12:00:57.714 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already in. 2021-05-12T12:00:57.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:00:57.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2021-05-12T12:00:57.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:00:57.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:00:57.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:00:57.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:00:57.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:00:57.726 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:00:57.726 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:00:57.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:00:57.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:00:57.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:00:57.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:00:57.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:00:57.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:00:57.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:00:57.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:00:58.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:00:58.214 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:00:58.214 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:00:58.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:00:58.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:58.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:00:58.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=146028888068 2021-05-12T12:00:58.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 146028888068 2021-05-12T12:00:58.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888068' 2021-05-12T12:00:58.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:58.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:00:58.448 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:00:58.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:00:58.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:00:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:00:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:00:58.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:00:58.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822659 2021-05-12T12:00:58.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822659 2021-05-12T12:00:58.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-146028888068 2-154618822659' 2021-05-12T12:00:58.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:00:58.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-146028888068 2021-05-12T12:00:58.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:00:58.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:00:58.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-146028888068 2021-05-12T12:00:58.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:00:58.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=146028888068 2021-05-12T12:00:58.582 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 146028888068 2021-05-12T12:00:58.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2021-05-12T12:00:58.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:00:58.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888066 -lt 146028888068 2021-05-12T12:00:58.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:00:59.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:00:59.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:01:00.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888068 -lt 146028888068 2021-05-12T12:01:00.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:01:00.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2021-05-12T12:01:00.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:01:00.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:01:00.315 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-154618822659 2021-05-12T12:01:00.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:01:00.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822659 2021-05-12T12:01:00.316 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 154618822659 2021-05-12T12:01:00.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2021-05-12T12:01:00.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:01:00.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822659 -lt 154618822659 2021-05-12T12:01:00.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:01:00.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:00.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:01.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:01:01.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:01.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:01.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:01.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:01.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:01.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:01.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:01.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:01.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:01.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:01.554 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:02.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:02.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:01:02.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:01:02.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:01:02.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:02.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:01:02.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:02.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:02.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:02.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:02.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:02.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:02.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:02.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:02.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:02.478 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:02.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:02.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:02.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:02.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:02.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:02.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:02.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:02.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:02.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:02.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:02.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:03.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:03.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:03.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:01:03.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:03.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:01:03.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:03.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:03.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:03.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:04.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:04.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:04.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:04.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:04.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:04.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:04.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:04.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:04.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:04.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:04.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:04.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:04.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:04.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:04.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:04.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:01:04.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:04.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:01:05.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:05.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:05.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:05.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:05.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:05.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:05.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:05.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:05.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:05.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:05.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:05.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:06.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:06.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:06.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:06.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:06.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:06.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:06.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:06.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:06.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:06.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:06.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:06.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:01:06.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:06.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:01:07.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:07.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:07.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:07.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:07.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:07.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:07.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:07.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:07.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:07.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:07.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:07.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:08.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:08.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:08.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:08.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:08.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:08.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:08.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:08.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:08.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:08.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:08.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:01:08.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:08.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:01:10.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:10.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:10.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:10.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:10.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:10.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:10.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:10.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:10.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:10.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:10.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:10.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:11.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:11.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:11.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:11.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:11.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:11.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:11.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:11.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:11.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:11.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:11.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:11.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:01:11.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:11.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:01:14.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:14.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:14.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:14.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:14.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:14.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:14.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:14.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:15.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:15.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:15.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:15.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:15.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:15.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:15.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:15.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:15.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:15.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:15.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:15.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:15.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:16.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:16.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:16.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:01:16.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:16.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:01:22.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:22.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:22.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:22.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:22.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:22.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:22.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:22.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:23.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:23.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:23.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:23.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:23.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:23.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:23.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:23.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:23.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:23.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:23.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:23.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:23.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:23.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:23.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:23.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:01:23.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:01:23.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:01:31.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:01:31.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:01:31.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:01:31.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:01:31.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:01:31.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:01:31.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:01:31.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:01:31.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:01:31.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:01:31.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:01:31.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:01:32.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:01:32.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:01:32.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:01:32.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:01:32.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:01:32.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:01:32.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:01:32.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:01:32.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:01:32.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:01:32.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:01:32.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:01:32.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:01:33.050 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:01:33.050 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "70b5ba01-c156-440d-951a-0542e01714b7", 2021-05-12T12:01:33.050 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:01:33.051 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:01:33.051 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:01:33.046878+0000", 2021-05-12T12:01:33.051 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:01:33.051 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:01:33.051 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:01:33.052 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:01:33.052 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:01:33.052 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:01:33.052 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:01:33.053 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:01:33.053 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:01:33.053 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.053 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:01:33.053 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.054 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:01:33.054 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.054 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.054 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:01:33.055 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.055 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:01:33.055 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:01:33.055 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:01:33.055 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:01:33.056 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:01:33.056 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.056 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:01:33.056 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:01:33.057 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.057 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:01:33.057 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:01:33.057 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:01:33.057 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:01:33.058 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:01:33.058 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.058 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:01:33.058 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.059 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:01:33.059 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.059 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.059 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:01:33.059 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.060 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:01:33.060 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:01:33.060 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:01:33.060 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:01:33.061 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:01:33.061 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.061 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:01:33.061 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.061 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:01:33.062 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.062 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.062 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:01:33.062 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.062 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:01:33.063 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:01:33.063 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:01:33.063 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:01:33.063 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:01:33.064 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.064 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:01:33.064 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.064 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:01:33.064 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.065 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.065 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:01:33.065 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.065 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.066 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:01:33.066 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.066 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.066 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:01:33.067 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.067 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:01:33.067 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:01:33.067 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:01:33.067 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:01:33.068 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:01:33.068 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T11:59:34.811314Z", 2021-05-12T12:01:33.068 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T11:59:34.811314Z", 2021-05-12T12:01:33.068 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:01:33.069 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:01:33.070 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:01:33.070 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:01:33.070 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:01:33.070 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:01:33.071 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:01:33.071 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:01:33.071 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:01:33.071 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:01:33.071 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:01:33.072 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.072 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:01:33.072 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.072 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:01:33.073 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.073 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:01:33.073 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:01:33.073 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:01:33.073 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.074 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.074 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.074 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:01:33.074 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:01:33.074 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.075 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.075 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.075 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:01:33.075 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:01:33.075 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:01:33.076 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:01:33.076 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:01:33.076 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.076 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.076 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.077 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:01:33.077 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:01:33.077 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.077 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:01:33.077 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 39, 2021-05-12T12:01:33.078 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:01:33.078 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T11:59:35.710247+0000", 2021-05-12T12:01:33.078 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:00:57.790009+0000", 2021-05-12T12:01:33.078 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:00:56.559968+0000", 2021-05-12T12:01:33.078 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:00:57.515049+0000", 2021-05-12T12:01:33.079 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:01:33.079 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:01:33.079 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:01:33.079 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:01:33.079 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:01:33.080 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:01:33.080 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:01:33.080 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:01:33.080 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:01:33.080 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:01:33.081 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.081 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 9, 2021-05-12T12:01:33.081 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:01:33.081 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:01:33.081 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:01:33.082 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:01:33.082 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:01:33.082 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:01:33.082 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:01:33.082 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:01:33.083 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:01:33.083 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:01:33.083 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.083 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:01:33.083 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:01:33.084 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:00:06.217856+0000", 2021-05-12T12:01:33.084 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 1, 2021-05-12T12:01:33.084 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool", 2021-05-12T12:01:33.084 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:01:33.084 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:01:33.085 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:01:33.085 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 0, 2021-05-12T12:01:33.085 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:01:33.085 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:01:33.085 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:01:33.086 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:01:33.086 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:01:33.086 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:01:33.086 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 1, 2021-05-12T12:01:33.086 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:01:33.087 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 1, 2021-05-12T12:01:33.087 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 1, 2021-05-12T12:01:33.087 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 1, 2021-05-12T12:01:33.087 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:01:33.087 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:01:33.088 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:01:33.088 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:01:33.088 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:01:33.088 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:01:33.088 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:01:33.089 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.089 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "27", 2021-05-12T12:01:33.089 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:01:33.089 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:01:33.089 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:01:33.090 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:01:33.090 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:01:33.090 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 0, 2021-05-12T12:01:33.090 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 0, 2021-05-12T12:01:33.090 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [], 2021-05-12T12:01:33.091 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:01:33.091 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:01:33.091 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:01:33.091 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:01:33.092 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:01:33.093 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:01:33.093 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:01:33.093 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:01:33.093 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:01:33.094 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:01:33.094 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "", 2021-05-12T12:01:33.094 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:01:33.094 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:01:33.094 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.095 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:01:33.095 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:01:33.095 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:01:33.095 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:01:33.095 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:01:33.096 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:01:33.096 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:01:33.096 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:01:33.096 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 0, 2021-05-12T12:01:33.096 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:01:33.097 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:01:33.097 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:01:33.097 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:01:33.097 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.097 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.098 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:01:33.098 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.098 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:01:33.098 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "ae18c233-dcbd-4a25-9ded-fc69c9202457", 2021-05-12T12:01:33.098 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:01:33.099 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:01:33.099 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:01:33.099 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:01:33.099 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:01:33.099 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 27, 2021-05-12T12:01:33.100 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 34, 2021-05-12T12:01:33.100 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 38, 2021-05-12T12:01:33.100 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 28, 2021-05-12T12:01:33.100 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:01:33.100 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:01:33.101 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.101 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.101 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.101 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:01:33.102 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.102 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.102 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.102 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.102 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:01:33.103 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.103 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.103 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.103 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.103 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:01:33.104 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.104 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.104 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.104 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:01:33.104 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.105 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.106 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.106 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.106 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:01:33.106 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.106 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.107 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.107 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:01:33.107 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.107 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.107 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.108 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.108 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:01:33.108 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.108 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.108 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.109 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.109 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:01:33.109 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.109 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.109 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.110 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:01:33.110 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.110 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.110 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.110 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.111 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:01:33.111 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 37583 2021-05-12T12:01:33.111 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.111 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.112 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.112 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/37583", 2021-05-12T12:01:33.112 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/37583", 2021-05-12T12:01:33.112 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/37583", 2021-05-12T12:01:33.112 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/37583", 2021-05-12T12:01:33.113 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:01:33.113 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:01:33.113 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:01:33.113 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.113 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.114 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.114 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:01:33.114 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "1ccf09c6-fc95-4724-bf58-2a61dee625e6", 2021-05-12T12:01:33.114 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:01:33.115 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:01:33.115 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:01:33.115 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:01:33.115 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:01:33.115 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:01:33.116 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 9, 2021-05-12T12:01:33.116 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 21, 2021-05-12T12:01:33.116 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 29, 2021-05-12T12:01:33.116 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:01:33.116 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.117 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.118 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.118 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.118 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:01:33.118 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.118 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.119 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.119 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.119 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:01:33.119 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.119 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.120 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.120 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:01:33.120 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.120 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.120 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.121 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.121 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:01:33.121 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.121 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.121 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.122 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.122 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:01:33.122 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.122 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.123 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.123 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:01:33.123 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.123 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.123 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.124 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.124 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:01:33.124 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.124 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.124 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.125 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.125 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:01:33.125 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.125 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.125 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.126 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:01:33.126 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.126 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.126 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.126 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.127 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:01:33.127 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 32171 2021-05-12T12:01:33.127 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.127 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.127 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.128 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/32171", 2021-05-12T12:01:33.128 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/32171", 2021-05-12T12:01:33.128 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/32171", 2021-05-12T12:01:33.128 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/32171", 2021-05-12T12:01:33.128 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:01:33.129 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:01:33.129 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.129 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.129 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.129 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:01:33.130 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "7537394c-b095-4135-98fc-d1a8fc5a0322", 2021-05-12T12:01:33.130 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:01:33.130 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:01:33.130 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:01:33.130 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:01:33.131 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 14, 2021-05-12T12:01:33.131 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 27, 2021-05-12T12:01:33.131 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 36, 2021-05-12T12:01:33.131 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:01:33.131 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 30, 2021-05-12T12:01:33.132 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:01:33.132 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:01:33.132 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.132 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.132 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.133 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:01:33.133 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.133 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.133 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.133 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.134 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:01:33.134 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.134 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.134 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.134 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.135 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:01:33.135 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.135 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.135 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.135 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:01:33.136 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.136 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.136 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.136 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.136 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:01:33.137 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.137 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.137 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.137 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.137 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:01:33.138 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.138 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.138 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.138 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:01:33.138 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.139 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.140 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.140 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.140 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:01:33.140 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:01:33.140 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.141 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:01:33.141 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:01:33.141 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.141 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.141 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.142 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:01:33.142 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:01:33.142 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 38166 2021-05-12T12:01:33.142 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.143 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.143 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.143 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/38166", 2021-05-12T12:01:33.143 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/38166", 2021-05-12T12:01:33.143 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38166", 2021-05-12T12:01:33.144 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38166", 2021-05-12T12:01:33.144 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:01:33.144 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:01:33.144 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.145 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:01:33.146 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:00:45.533166+0000", 2021-05-12T12:01:33.146 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:01:33.146 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:01:33.146 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:01:33.147 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:01:33.147 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T11:59:45.462237+0000", 2021-05-12T12:01:33.147 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:01:33.147 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.148 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.148 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:01:33.148 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:00:46.064564+0000", 2021-05-12T12:01:33.148 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:01:33.148 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:01:33.149 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:01:33.149 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:01:33.149 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T11:59:54.870113+0000", 2021-05-12T12:01:33.149 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:01:33.149 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.150 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.150 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:01:33.150 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:00:46.575177+0000", 2021-05-12T12:01:33.150 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:01:33.150 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:01:33.151 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:01:33.151 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:01:33.151 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:00:04.098375+0000", 2021-05-12T12:01:33.151 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:01:33.152 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.152 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.152 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:01:33.152 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:01:33.152 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:01:33.153 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:01:33.153 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:01:33.153 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:01:33.153 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:01:33.153 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:01:33.154 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:01:33.154 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:01:33.154 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:01:33.154 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.155 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.155 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:01:33.155 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:01:33.155 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:01:33.155 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:01:33.156 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:01:33.156 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:01:33.156 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:01:33.156 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:01:33.156 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:01:33.157 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:01:33.157 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:01:33.157 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.157 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.157 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:01:33.158 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.158 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:01:33.158 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:01:33.158 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/37583,v1:127.0.0.1:6805/37583]", 2021-05-12T12:01:33.159 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:01:33.159 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:01:33.159 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:01:33.159 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:01:33.159 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:01:33.160 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:01:33.160 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:01:33.160 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:01:33.160 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:01:33.161 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:01:33.161 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:01:33.161 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:01:33.161 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:01:33.161 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:01:33.162 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:01:33.162 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:01:33.162 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:01:33.162 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:01:33.163 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:01:33.163 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:01:33.163 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:01:33.163 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:01:33.163 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:01:33.164 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:01:33.164 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:01:33.164 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/37583,v1:127.0.0.1:6803/37583]", 2021-05-12T12:01:33.164 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:01:33.165 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/37583,v1:127.0.0.1:6809/37583]", 2021-05-12T12:01:33.165 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/37583,v1:127.0.0.1:6807/37583]", 2021-05-12T12:01:33.165 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:01:33.165 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:01:33.165 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:01:33.166 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:01:33.166 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:01:33.166 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:01:33.166 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:01:33.167 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:01:33.167 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:01:33.167 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:01:33.167 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:01:33.167 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:01:33.168 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:01:33.168 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.168 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.168 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:01:33.168 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:01:33.169 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/32171,v1:127.0.0.1:6813/32171]", 2021-05-12T12:01:33.169 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:01:33.169 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:01:33.169 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:01:33.170 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:01:33.170 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:01:33.170 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:01:33.170 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:01:33.170 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:01:33.171 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:01:33.171 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:01:33.171 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:01:33.171 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:01:33.171 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:01:33.172 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:01:33.172 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:01:33.172 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:01:33.172 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:01:33.173 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:01:33.173 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:01:33.173 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:01:33.173 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:01:33.173 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:01:33.174 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:01:33.174 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:01:33.174 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:01:33.174 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/32171,v1:127.0.0.1:6811/32171]", 2021-05-12T12:01:33.175 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:01:33.175 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/32171,v1:127.0.0.1:6817/32171]", 2021-05-12T12:01:33.175 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/32171,v1:127.0.0.1:6815/32171]", 2021-05-12T12:01:33.175 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:01:33.175 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:01:33.176 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:01:33.176 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:01:33.176 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:01:33.176 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:01:33.176 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:01:33.177 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:01:33.177 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:01:33.177 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:01:33.177 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:01:33.177 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:01:33.178 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:01:33.178 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.178 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.178 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:01:33.179 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:01:33.179 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/38166,v1:127.0.0.1:6813/38166]", 2021-05-12T12:01:33.179 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:01:33.179 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:01:33.179 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:01:33.180 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:01:33.180 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:01:33.180 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:01:33.180 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:01:33.180 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:01:33.181 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:01:33.181 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:01:33.181 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:01:33.181 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:01:33.182 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:01:33.182 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:01:33.182 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:01:33.182 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:01:33.182 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:01:33.183 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:01:33.183 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:01:33.183 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:01:33.183 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:01:33.183 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:01:33.184 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:01:33.184 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:01:33.184 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:01:33.184 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/38166,v1:127.0.0.1:6811/38166]", 2021-05-12T12:01:33.185 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:01:33.185 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38166,v1:127.0.0.1:6817/38166]", 2021-05-12T12:01:33.185 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38166,v1:127.0.0.1:6815/38166]", 2021-05-12T12:01:33.185 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:01:33.185 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:01:33.186 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:01:33.186 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:01:33.186 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:01:33.187 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:01:33.187 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:01:33.187 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:01:33.187 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:01:33.187 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:01:33.188 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:01:33.188 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:01:33.188 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:01:33.188 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.188 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.189 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:01:33.189 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:01:33.189 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:01:33.189 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:01:33.190 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.190 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:01:33.190 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.190 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:01:33.191 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 34 2021-05-12T12:01:33.191 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.191 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.191 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:01:33.191 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 9 2021-05-12T12:01:33.192 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.192 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.192 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:01:33.192 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 37 2021-05-12T12:01:33.192 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.193 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.193 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.193 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:01:33.193 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 39, 2021-05-12T12:01:33.193 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:01:33.194 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:01:33.194 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.194 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:01:33.194 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:01:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:01:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:01:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:01:33.196 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:01:33.196 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.196 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.196 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:01:33.197 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:01:33.197 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:01:33.197 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.197 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.198 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:01:33.198 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.198 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:01:33.198 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:01:33.198 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.199 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.199 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:01:33.199 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:01:33.199 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.200 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.200 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:01:33.200 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:01:33.200 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.200 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.201 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:01:33.201 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:01:33.201 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.201 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.202 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:01:33.202 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:01:33.202 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.202 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.203 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:01:33.203 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:01:33.203 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.203 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.204 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:01:33.204 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:01:33.204 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.204 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.205 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:01:33.205 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:01:33.205 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.205 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:01:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:01:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:01:33.207 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:01:33.207 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.207 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.207 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:01:33.208 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:01:33.208 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.208 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.208 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:01:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:01:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:01:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.210 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:01:33.210 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:01:33.210 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:01:33.210 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:01:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:01:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:01:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:01:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.212 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:01:33.212 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.212 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:01:33.212 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:01:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:01:33.214 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:01:33.214 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:01:33.214 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.214 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:01:33.215 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:01:33.215 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:01:33.215 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.215 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:01:33.215 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:01:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.217 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:01:33.217 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:01:33.217 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:01:33.217 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:01:33.217 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.218 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:01:33.218 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:01:33.218 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:01:33.218 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.219 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:01:33.219 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.219 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:01:33.219 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.219 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:01:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:01:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.221 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:01:33.221 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.221 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:01:33.221 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.222 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.222 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.222 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.222 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:01:33.222 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:01:33.223 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:01:33.223 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:01:33.223 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:01:33.223 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:01:33.224 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:01:33.224 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:01:33.224 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.224 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:01:33.224 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.225 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:01:33.225 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.225 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.225 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:01:33.225 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.226 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:01:33.226 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.226 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.226 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:01:33.227 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:01:33.227 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:01:33.227 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.227 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.227 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.228 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.228 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:01:33.228 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.228 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:01:33.228 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:01:33.229 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:01:33.229 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:01:33.229 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:01:33.229 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:01:33.230 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:01:33.230 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.230 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:01:33.230 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:01:33.230 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:01:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:01:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:01:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:01:33.232 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.232 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.232 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:01:33.232 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.233 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:01:33.233 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.233 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.233 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:01:33.233 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:01:33.234 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:01:33.234 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:01:33.234 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:01:33.234 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:01:33.234 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:01:33.235 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:01:33.235 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:01:33.235 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:01:33.235 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:01:33.236 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:01:33.236 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:01:33.236 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:01:33.236 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:01:33.236 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 0, 2021-05-12T12:01:33.237 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:01:33.237 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:01:33.237 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:01:33.237 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:01:33.237 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:01:33.238 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.238 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:01:33.238 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.238 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:01:33.239 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:01:33.239 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:01:33.239 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:01:33.239 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:01:33.239 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:01:33.240 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:01:33.240 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:01:33.240 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:01:33.240 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:01:33.240 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:01:33.241 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:01:33.241 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:01:33.241 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:01:33.241 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:01:33.242 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:01:33.242 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.242 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.242 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:01:33.242 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:01:33.243 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:01:33.243 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.243 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:01:33.243 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:01:33.243 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.244 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:01:33.244 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:01:33.244 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:01:33.244 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:01:33.245 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:01:33.245 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:01:33.245 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.245 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:01:33.245 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 1, 2021-05-12T12:01:33.246 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:01:33.246 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:01:33.246 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:01:33.246 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:01:33.246 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:01:33.247 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:01:33.247 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:01:33.247 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:01:33.247 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:01:33.248 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:01:33.248 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:01:33.248 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:01:33.248 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:01:33.248 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:01:33.249 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:01:33.249 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:01:33.249 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:01:33.249 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:01:33.249 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:01:33.250 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:01:33.250 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:01:33.250 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:01:33.250 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:01:33.251 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:01:33.251 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:01:33.251 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:01:33.251 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:01:33.251 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:01:33.252 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:01:33.252 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:01:33.252 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:01:33.252 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:01:33.252 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:01:33.253 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:01:33.253 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:01:33.253 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:01:33.253 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:01:33.253 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:01:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:01:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:01:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:01:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:01:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:01:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:01:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:01:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:01:33.256 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:01:33.256 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:01:33.256 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:01:33.256 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:01:33.256 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:01:33.257 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:01:33.257 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:01:33.257 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:01:33.257 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:01:33.258 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.258 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 16, 2021-05-12T12:01:33.258 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 16, 2021-05-12T12:01:33.258 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:01:33.258 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:01:33.259 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:01:33.259 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.259 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:01:33.259 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:01:33.260 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:01:33.260 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:01:33.260 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:01:33.260 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:01:33.260 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:01:33.261 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:01:33.261 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18412, 2021-05-12T12:01:33.261 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2064, 2021-05-12T12:01:33.261 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:01:33.261 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:01:33.262 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554388, 2021-05-12T12:01:33.262 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:01:33.262 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:01:33.262 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103693312, 2021-05-12T12:01:33.263 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:01:33.263 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2113536, 2021-05-12T12:01:33.263 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 407137, 2021-05-12T12:01:33.263 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:01:33.263 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:01:33.264 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:01:33.264 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:01:33.264 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:01:33.264 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.264 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:01:33.265 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:01:33.265 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:01:33.265 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 1, 2021-05-12T12:01:33.265 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:01:33.266 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:01:33.266 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:01:33.266 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.266 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:01:33.266 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 110, 2021-05-12T12:01:33.267 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 110, 2021-05-12T12:01:33.267 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 110000000, 2021-05-12T12:01:33.267 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 110000000 2021-05-12T12:01:33.267 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.267 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:01:33.268 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:01:33.268 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.268 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:01:33.268 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:01:33.269 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:01:33.269 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:01:33.269 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:01:33.269 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:01:33.269 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:01:33.270 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:01:33.270 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:01:33.270 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18412, 2021-05-12T12:01:33.270 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2064, 2021-05-12T12:01:33.271 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:01:33.271 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:01:33.271 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554388, 2021-05-12T12:01:33.271 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:01:33.271 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:01:33.272 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103693312, 2021-05-12T12:01:33.272 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:01:33.272 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2113536, 2021-05-12T12:01:33.272 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 407137, 2021-05-12T12:01:33.273 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:01:33.273 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:01:33.273 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:01:33.273 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:01:33.273 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:01:33.274 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.274 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:01:33.274 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:01:33.274 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:01:33.274 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 1, 2021-05-12T12:01:33.275 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:01:33.275 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:01:33.275 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:01:33.275 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.276 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:01:33.276 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 110, 2021-05-12T12:01:33.276 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 110, 2021-05-12T12:01:33.276 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 110000000, 2021-05-12T12:01:33.276 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 110000000 2021-05-12T12:01:33.277 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.277 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:01:33.277 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:01:33.277 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.278 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.278 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:01:33.278 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.278 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:01:33.278 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:01:33.279 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:01:33.279 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:01:33.279 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:01:33.279 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:01:33.279 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:01:33.280 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:01:33.280 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:01:33.280 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:01:33.280 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:01:33.281 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:01:33.281 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:01:33.281 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:01:33.281 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:01:33.281 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:01:33.282 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:01:33.282 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:01:33.282 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:01:33.282 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:01:33.283 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:01:33.283 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:01:33.283 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:01:33.283 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:01:33.283 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:01:33.284 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:01:33.284 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:01:33.284 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:01:33.284 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:01:33.284 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:01:33.285 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:01:33.285 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:01:33.285 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:01:33.285 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:01:33.285 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:01:33.286 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:01:33.286 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:01:33.286 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:01:33.286 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:01:33.287 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:01:33.287 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:01:33.287 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:01:33.287 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:01:33.287 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.288 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:01:33.288 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:01:33.288 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:01:33.288 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:01:33.288 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1228800, 2021-05-12T12:01:33.289 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 144900, 2021-05-12T12:01:33.289 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:01:33.289 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:01:33.289 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:01:33.289 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:01:33.290 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:01:33.290 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.290 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 16, 2021-05-12T12:01:33.290 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 16, 2021-05-12T12:01:33.291 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:01:33.291 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:01:33.291 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:01:33.291 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.291 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.292 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:01:33.292 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.292 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:01:33.292 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 146028888074 2021-05-12T12:01:33.293 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.293 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.293 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:01:33.293 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 38654705681 2021-05-12T12:01:33.293 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.294 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.294 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:01:33.294 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 154618822665 2021-05-12T12:01:33.294 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.295 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.295 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:01:33.295 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.295 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:01:33.295 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 1 2021-05-12T12:01:33.296 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.296 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.296 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:01:33.296 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.296 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:01:33.297 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:01:33.297 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:01:33.297 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:01:33.297 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.298 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.298 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:01:33.298 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:01:33.298 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:01:33.298 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:01:33.299 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.299 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.299 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:01:33.299 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 1, 2021-05-12T12:01:33.299 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:01:33.300 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:01:33.300 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.300 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.300 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:01:33.300 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:01:33.301 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:01:33.301 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:01:33.301 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.301 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:01:33.301 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:01:33.302 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 4, 2021-05-12T12:01:33.302 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:00:56.941443+0000", 2021-05-12T12:01:33.302 INFO:tasks.workunit.client.0.smithi090.stdout: "services": { 2021-05-12T12:01:33.302 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": { 2021-05-12T12:01:33.303 INFO:tasks.workunit.client.0.smithi090.stdout: "daemons": { 2021-05-12T12:01:33.303 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": "", 2021-05-12T12:01:33.303 INFO:tasks.workunit.client.0.smithi090.stdout: "0": { 2021-05-12T12:01:33.303 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:01:33.304 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:01:33.304 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:01:33.304 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:01:33.304 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:01:33.304 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:01:33.305 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.305 INFO:tasks.workunit.client.0.smithi090.stdout: "2": { 2021-05-12T12:01:33.305 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:01:33.305 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:01:33.305 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:01:33.306 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:01:33.306 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:01:33.306 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:01:33.306 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.307 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.307 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.307 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.307 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:01:33.307 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:01:33.308 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:01:33.308 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 358, 2021-05-12T12:01:33.308 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:01:33.308 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:01:33.309 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:01:33.309 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:01:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:report 3474182418 2021-05-12T12:01:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:01:33.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2021-05-12T12:01:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2021-05-12T12:01:33.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2021-05-12T12:01:33.311 INFO:tasks.workunit.client.0.smithi090.stdout:writing non-divergent object existing_4 2021-05-12T12:01:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:33.399 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:33.400 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 27'100 39:121 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:33.400 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:33.400 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:33.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:01:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2021-05-12T12:01:33.523 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery 2021-05-12T12:01:33.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:33.865 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:33.865 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 27'100 39:121 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:33.866 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:33.866 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:33.867 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:33.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T12:01:33.877 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2021-05-12T12:01:33.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:01:33.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:01:33.878 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:01:33.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:01:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T12:01:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:01:33.972 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:01:33.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T12:01:33.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2021-05-12T12:01:33.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:01:33.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:01:33.982 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:33.983 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:33.983 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:01:33.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:01:33.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T12:01:33.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:01:34.076 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:01:34.084 INFO:tasks.workunit.client.0.smithi090.stdout:revive divergent 1 2021-05-12T12:01:34.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2021-05-12T12:01:34.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:34.422 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:34.423 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 27'100 39:121 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:34.423 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:34.423 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:34.424 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:34.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2021-05-12T12:01:35.095 INFO:tasks.workunit.client.0.smithi090.stderr:noup is set 2021-05-12T12:01:35.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2021-05-12T12:01:35.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:01:35.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:01:35.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:01:35.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:01:35.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:01:35.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:01:35.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:01:35.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:01:35.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:01:35.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:01:35.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:01:35.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:01:35.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:01:35.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:01:35.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:01:35.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:35.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:35.113 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:35.114 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:35.114 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:01:35.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:01:35.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:01:35.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:01:35.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:01:35.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:01:35.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:01:35.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:01:35.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:35.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:01:35.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:01:35.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:01:35.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:01:35.117 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:01:35.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:35.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:01:35.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:01:35.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:01:35.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:01:35.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:01:35.134 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:35.132+0000 7f3169a240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:35.136 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:35.134+0000 7f3169a240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:35.137 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:35.135+0000 7f3169a240c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:35.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:01:35.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2021-05-12T12:01:35.654 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:35.652+0000 7f3169a240c0 -1 Falling back to public interface 2021-05-12T12:01:38.291 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:38.288+0000 7f3169a240c0 -1 osd.1 27 log_to_monitors {default=true} 2021-05-12T12:01:40.474 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery divergent 2021-05-12T12:01:40.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2021-05-12T12:01:40.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:40.823 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:40.824 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:40.824 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:40.824 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:40.825 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:40.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2021-05-12T12:01:40.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:01:40.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:01:40.836 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:01:40.836 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:40.837 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:01:40.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:01:40.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2021-05-12T12:01:40.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:01:40.930 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:01:40.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2021-05-12T12:01:41.616 INFO:tasks.workunit.client.0.smithi090.stderr:noup is unset 2021-05-12T12:01:41.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2021-05-12T12:01:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:01:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:41.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:41.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:41.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:41.627 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:01:41.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:41.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:01:42.047 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/37583,v1:127.0.0.1:6803/37583] [v2:127.0.0.1:6804/37583,v1:127.0.0.1:6805/37583] exists,up ae18c233-dcbd-4a25-9ded-fc69c9202457 2021-05-12T12:01:42.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:42.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:42.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:42.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2021-05-12T12:01:42.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:42.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:01:42.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:42.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:42.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:42.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:42.051 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:01:42.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:42.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:01:42.413 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [9,27) [v2:127.0.0.1:6818/39893,v1:127.0.0.1:6819/39893] [v2:127.0.0.1:6820/39893,v1:127.0.0.1:6821/39893] exists,up 1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T12:01:42.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:42.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:42.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:42.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2021-05-12T12:01:42.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:42.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:01:42.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:42.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:42.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:42.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:42.417 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:01:42.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:42.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:01:42.777 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/38166,v1:127.0.0.1:6811/38166] [v2:127.0.0.1:6812/38166,v1:127.0.0.1:6813/38166] exists,up 7537394c-b095-4135-98fc-d1a8fc5a0322 2021-05-12T12:01:42.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:42.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:42.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:42.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:43.119 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:43.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:43.120 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:43.120 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:43.121 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:43.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2021-05-12T12:01:43.130 INFO:tasks.workunit.client.0.smithi090.stdout:wait for peering 2021-05-12T12:01:43.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:43.477 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:43.477 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:00:58.025707+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:43.477 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:43.478 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:43.478 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:43.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2021-05-12T12:01:54.133 INFO:tasks.workunit.client.0.smithi090.stdout:killing divergent 1 2021-05-12T12:01:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2021-05-12T12:01:54.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:54.484 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:54.484 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T12:01:42.249306+0000 39'101 45:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:54.485 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:54.485 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:54.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:01:54.496 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:01:54.496 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:01:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:01:54.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:01:54.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:01:54.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:01:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:reviving divergent 1 2021-05-12T12:01:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2021-05-12T12:01:54.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2021-05-12T12:01:54.943 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:01:54.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T12:01:42.249306+0000 39'101 45:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:01:54.943 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:01:54.944 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:01:54.944 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:01:54.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2021-05-12T12:01:54.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:01:54.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:01:54.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:01:54.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:01:54.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:01:54.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:01:54.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:01:54.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:01:54.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:01:54.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:01:54.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:01:54.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:01:54.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:01:54.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:01:54.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:01:54.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:01:54.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:01:54.961 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:01:54.961 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:01:54.961 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:01:54.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:01:54.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:01:54.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:01:54.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:01:54.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:01:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:01:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:01:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:01:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:01:54.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:01:54.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:01:54.964 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:01:54.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:01:54.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:01:54.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:01:54.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:01:54.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:01:54.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:01:54.984 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:54.982+0000 7fdc6eaf80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:54.986 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:54.984+0000 7fdc6eaf80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:54.988 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:54.986+0000 7fdc6eaf80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:01:55.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:01:55.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:01:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:01:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:01:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:01:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:01:55.325 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:01:55.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:01:55.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:01:55.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:01:55.503 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:55.500+0000 7fdc6eaf80c0 -1 Falling back to public interface 2021-05-12T12:01:55.702 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 29 last_clean_interval [9,27) [v2:127.0.0.1:6818/39893,v1:127.0.0.1:6819/39893] [v2:127.0.0.1:6820/39893,v1:127.0.0.1:6821/39893] exists,up 1ccf09c6-fc95-4724-bf58-2a61dee625e6 2021-05-12T12:01:55.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:01:55.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:01:55.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:01:55.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2021-05-12T12:01:58.012 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:01:58.010+0000 7fdc6eaf80c0 -1 osd.1 45 log_to_monitors {default=true} 2021-05-12T12:02:15.704 INFO:tasks.workunit.client.0.smithi090.stdout:allowing recovery 2021-05-12T12:02:15.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2021-05-12T12:02:15.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2021-05-12T12:02:16.053 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:16.054 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T12:02:03.281612+0000 45'102 50:152 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:02:16.054 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:02:16.054 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:16.055 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:02:16.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:02:16.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:02:16.188 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:02:16.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:02:16.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:02:16.319 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:02:16.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T12:02:16.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:02:16.451 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:02:16.459 INFO:tasks.workunit.client.0.smithi090.stdout:reading divergent objects 2021-05-12T12:02:16.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2021-05-12T12:02:16.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2021-05-12T12:02:16.801 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:02:16.802 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T12:02:03.281612+0000 45'102 50:152 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:00:06.307794+0000 0'0 2021-05-12T12:00:06.307794+0000 0 2021-05-12T12:02:16.802 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:02:16.802 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:02:16.803 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:02:16.813 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2021-05-12T12:02:16.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2021-05-12T12:02:16.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:02:16.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:02:16.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:02:16.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:02:16.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:02:16.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:02:16.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:02:16.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:02:16.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2021-05-12T12:02:16.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T12:02:16.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:02:16.951 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4532.0:1 2021-05-12T12:00:34.161125+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:02:16.951 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4532.0:1 2021-05-12T12:00:34.161125+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:02:16.951 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:02:16.952 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'101 2021-05-12T12:02:16.952 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T12:02:16.952 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:02:16.952 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 27'101 2021-05-12T12:02:16.953 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4532.0:1 2021-05-12T12:00:34.161125+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:02:16.953 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:01:42.122+0000 7f3145919700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T12:02:16.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T12:02:16.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:02:16.956 INFO:tasks.workunit.client.0.smithi090.stdout:success 2021-05-12T12:02:16.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2021-05-12T12:02:16.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2021-05-12T12:02:16.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:02:16.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:02:17.416 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' does not exist 2021-05-12T12:02:17.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2021-05-12T12:02:17.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:02:17.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:02:17.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:02:17.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:02:17.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:02:17.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:02:17.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:02:17.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:02:17.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:02:17.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:02:17.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:02:17.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:02:17.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:02:17.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:02:17.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:02:17.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:02:17.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:02:17.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:02:17.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:02:17.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:02:17.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:02:17.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:02:17.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:02:17.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:02:17.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:02:17.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:02:17.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:02:17.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:02:17.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:02:17.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:02:17.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:17.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:17.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:02:17.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:02:17.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:02:17.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:02:17.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:02:17.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:02:17.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:02:17.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:02:17.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:02:17.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:02:17.560 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:02:17.560 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:02:17.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:02:17.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:02:17.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:02:17.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:02:17.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:02:17.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:02:17.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:02:17.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:02:17.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:02:17.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:02:17.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:02:17.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:02:17.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:02:17.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:02:17.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:02:17.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:02:17.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:02:17.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:02:17.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:17.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:17.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:02:17.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:02:17.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:02:17.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:02:17.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:02:17.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:17.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:17.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29659 2021-05-12T12:02:17.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:02:17.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:02:17.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:02:17.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:02:17.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2021-05-12T12:02:17.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2021-05-12T12:02:17.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2021-05-12T12:02:17.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2021-05-12T12:02:17.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2021-05-12T12:02:17.574 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2021-05-12T12:02:17.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2021-05-12T12:02:17.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2021-05-12T12:02:17.575 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:02:17.575 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:02:17.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2021-05-12T12:02:17.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:02:17.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:02:17.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:02:17.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:02:17.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:02:17.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:02:17.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:02:17.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:17.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:17.923 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:17.923 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:17.924 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:17.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:17.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:02:18.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:02:18.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:02:18.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:02:18.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:02:18.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:02:18.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:02:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:02:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:02:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:02:18.186 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:02:18.186 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:18.186 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:18.186 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:02:18.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:02:18.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get fsid 2021-05-12T12:02:18.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:02:18.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:02:18.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:02:18.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:02:18.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:02:18.281 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:02:18.281 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:02:18.281 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:02:18.282 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:02:18.282 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:18.282 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:18.282 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:02:18.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:02:18.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get mon_host 2021-05-12T12:02:18.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2021-05-12T12:02:18.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:02:18.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:02:18.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:02:18.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:02:18.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:02:18.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:02:18.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:02:18.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:18.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:18.861 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:18.861 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:18.861 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:18.861 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:18.862 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:02:18.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:02:18.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:02:18.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2021-05-12T12:02:18.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:02:18.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:02:18.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:02:18.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:02:18.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:02:18.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:02:18.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:02:18.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:02:18.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:02:18.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:02:18.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:02:18.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:02:18.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:02:18.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:02:18.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:02:18.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:18.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:18.895 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:18.895 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:18.896 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:18.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:18.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:18.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:02:18.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:02:18.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:02:18.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:02:18.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:02:18.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:02:18.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:02:18.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:18.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:02:18.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:02:18.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:02:18.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:02:18.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:02:18.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 f2cb872d-ef34-4fe5-a813-b277592bb9a2' 2021-05-12T12:02:18.902 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:02:18.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:02:18.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDKw5tgrJImNhAAh+SVhb6Xgy47w1Fej+CYkw== 2021-05-12T12:02:18.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDKw5tgrJImNhAAh+SVhb6Xgy47w1Fej+CYkw=="}' 2021-05-12T12:02:18.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f2cb872d-ef34-4fe5-a813-b277592bb9a2 -i td/divergent-priors/0/new.json 2021-05-12T12:02:19.241 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:02:19.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:02:19.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKw5tgrJImNhAAh+SVhb6Xgy47w1Fej+CYkw== --osd-uuid f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:02:19.283 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.279+0000 7f76704600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:19.284 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.281+0000 7f76704600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:19.286 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.283+0000 7f76704600c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:19.287 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.284+0000 7f76704600c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:02:19.287 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.284+0000 7f76704600c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:02:19.288 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.284+0000 7f76704600c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:02:19.324 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:19.321+0000 7f76704600c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:02:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:02:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:02:23.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:02:23.974 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:02:23.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:02:24.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:02:24.452 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:02:24.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:24.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:02:24.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:02:24.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:02:24.480 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:24.477+0000 7f597c3b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:24.482 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:24.479+0000 7f597c3b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:24.483 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:24.480+0000 7f597c3b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:02:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:02:24.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:02:24.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:02:24.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:02:24.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:24.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:02:24.814 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:02:24.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:24.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:02:25.000 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:24.997+0000 7f597c3b50c0 -1 Falling back to public interface 2021-05-12T12:02:25.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:26.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:26.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:26.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:02:26.177 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:02:26.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:26.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:02:26.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:27.181 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:27.179+0000 7f597c3b50c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:02:27.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:27.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:27.548 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:02:27.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:02:27.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:27.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:02:27.922 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42526,v1:127.0.0.1:6803/42526] [v2:127.0.0.1:6804/42526,v1:127.0.0.1:6805/42526] exists,up f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:02:27.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:02:27.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:02:27.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:02:27.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:02:27.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2021-05-12T12:02:27.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:02:27.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:02:27.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:02:27.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:02:27.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:02:27.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:02:27.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:02:27.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:02:27.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:02:27.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:02:27.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:02:27.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:02:27.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:02:27.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:02:27.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:02:27.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:27.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:27.931 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:27.932 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:27.932 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:27.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:27.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:27.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:02:27.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:02:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:02:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:02:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:02:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:02:27.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:02:27.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:02:27.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 9f1bd5b2-c83e-463c-b023-d48ebd49764a' 2021-05-12T12:02:27.935 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:02:27.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:02:27.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDTw5tgLl0yOBAAd9/fS1svWbPvkjytfgpNwQ== 2021-05-12T12:02:27.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDTw5tgLl0yOBAAd9/fS1svWbPvkjytfgpNwQ=="}' 2021-05-12T12:02:27.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9f1bd5b2-c83e-463c-b023-d48ebd49764a -i td/divergent-priors/1/new.json 2021-05-12T12:02:28.500 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:02:28.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:02:28.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDTw5tgLl0yOBAAd9/fS1svWbPvkjytfgpNwQ== --osd-uuid 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:02:28.542 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.540+0000 7fb3dae1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:28.547 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.544+0000 7fb3dae1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:28.549 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.546+0000 7fb3dae1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:28.549 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.547+0000 7fb3dae1e0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:02:28.549 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.547+0000 7fb3dae1e0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:02:28.550 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.547+0000 7fb3dae1e0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:02:28.622 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:28.620+0000 7fb3dae1e0c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:02:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:02:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:02:33.119 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:02:33.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:02:33.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:02:33.633 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:02:33.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:02:33.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:02:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:02:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:02:33.661 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:33.659+0000 7fbed362e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:33.663 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:33.661+0000 7fbed362e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:33.662+0000 7fbed362e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:33.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:02:33.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:02:33.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:02:33.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:02:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:02:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:33.999 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:02:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:02:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:02:34.181 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:34.178+0000 7fbed362e0c0 -1 Falling back to public interface 2021-05-12T12:02:34.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:35.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:35.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:35.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:02:35.378 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:02:35.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:35.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:02:35.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:36.355 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:36.353+0000 7fbed362e0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:02:36.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:36.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:36.752 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:02:36.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:02:36.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:36.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:02:37.118 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43911,v1:127.0.0.1:6811/43911] [v2:127.0.0.1:6812/43911,v1:127.0.0.1:6813/43911] exists,up 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:02:37.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:02:37.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:02:37.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:02:37.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T12:02:37.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2021-05-12T12:02:37.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:02:37.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:02:37.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:02:37.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:02:37.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:02:37.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:02:37.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:02:37.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:02:37.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:02:37.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:02:37.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:02:37.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:02:37.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:02:37.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:02:37.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:02:37.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:02:37.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:02:37.127 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:02:37.128 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:02:37.128 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:02:37.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:37.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:02:37.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:02:37.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:02:37.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:02:37.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:02:37.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:02:37.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:02:37.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:02:37.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:37.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:02:37.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:02:37.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:02:37.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:02:37.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:02:37.131 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:02:37.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0' 2021-05-12T12:02:37.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:02:37.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDdw5tghQNGCBAAOkQV+FNH9hieOfdox321XQ== 2021-05-12T12:02:37.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDdw5tghQNGCBAAOkQV+FNH9hieOfdox321XQ=="}' 2021-05-12T12:02:37.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 -i td/divergent-priors/2/new.json 2021-05-12T12:02:37.657 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:02:37.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:02:37.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdw5tghQNGCBAAOkQV+FNH9hieOfdox321XQ== --osd-uuid c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:02:37.698 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.696+0000 7fecb95610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:37.699 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.697+0000 7fecb95610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:37.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.699+0000 7fecb95610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:37.702 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.700+0000 7fecb95610c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:37.702 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.700+0000 7fecb95610c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:37.703 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.700+0000 7fecb95610c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:02:37.754 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:37.752+0000 7fecb95610c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:02:42.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:02:42.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:02:42.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:02:42.320 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:02:42.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:02:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:02:42.810 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:02:42.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:02:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:02:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:02:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:02:42.837 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:42.834+0000 7f146ebaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:42.838 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:42.836+0000 7f146ebaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:42.840 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:42.838+0000 7f146ebaa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:02:43.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:02:43.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:02:43.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:02:43.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:02:43.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:02:43.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:43.177 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:02:43.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:02:43.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:43.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:43.356 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:43.354+0000 7f146ebaa0c0 -1 Falling back to public interface 2021-05-12T12:02:43.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:44.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:44.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:44.567 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:02:44.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:02:44.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:44.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:44.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:02:45.577 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:02:45.575+0000 7f146ebaa0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:02:45.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:02:45.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:02:45.930 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:02:45.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:02:45.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:02:45.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:02:46.312 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/45296,v1:127.0.0.1:6819/45296] [v2:127.0.0.1:6820/45296,v1:127.0.0.1:6821/45296] exists,up c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:02:46.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:02:46.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:02:46.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:02:46.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2021-05-12T12:02:46.998 INFO:tasks.workunit.client.0.smithi090.stderr:noout is set 2021-05-12T12:02:47.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2021-05-12T12:02:47.759 INFO:tasks.workunit.client.0.smithi090.stderr:noin is set 2021-05-12T12:02:47.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2021-05-12T12:02:48.520 INFO:tasks.workunit.client.0.smithi090.stderr:nodown is set 2021-05-12T12:02:48.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2021-05-12T12:02:48.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T12:02:49.014 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' already exists 2021-05-12T12:02:49.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:02:50.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2021-05-12T12:02:50.736 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 3 2021-05-12T12:02:50.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2021-05-12T12:02:51.449 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 min_size to 2 2021-05-12T12:02:51.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2021-05-12T12:02:51.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:02:51.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:02:51.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:02:51.859 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:02:51.859 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:02:51.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:02:51.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:51.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:02:51.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T12:02:51.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T12:02:51.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T12:02:51.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:51.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:02:52.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T12:02:52.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T12:02:52.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2021-05-12T12:02:52.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:52.126 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:02:52.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T12:02:52.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T12:02:52.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509444' 2021-05-12T12:02:52.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:52.258 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T12:02:52.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:52.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:02:52.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T12:02:52.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:52.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T12:02:52.261 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836488 2021-05-12T12:02:52.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T12:02:52.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:52.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836486 -lt 21474836488 2021-05-12T12:02:52.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:02:53.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:02:53.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:53.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T12:02:53.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:53.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T12:02:53.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:53.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:02:53.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T12:02:53.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:53.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T12:02:53.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T12:02:53.996 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672965 2021-05-12T12:02:53.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672965 2021-05-12T12:02:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:54.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T12:02:54.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:02:54.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T12:02:54.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T12:02:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T12:02:54.360 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509444 2021-05-12T12:02:54.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:54.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T12:02:54.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2021-05-12T12:02:54.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:02:54.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:02:54.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:02:54.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:02:54.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:02:54.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:02:54.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:02:54.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:02:54.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:02:54.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:02:54.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:02:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:02:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:02:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:02:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:02:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:55.175 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:02:55.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T12:02:55.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T12:02:55.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T12:02:55.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:55.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:02:55.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T12:02:55.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T12:02:55.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672968' 2021-05-12T12:02:55.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:02:55.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:02:55.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509446 2021-05-12T12:02:55.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509446 2021-05-12T12:02:55.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672968 2-64424509446' 2021-05-12T12:02:55.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:55.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T12:02:55.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:55.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:02:55.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T12:02:55.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:55.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T12:02:55.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T12:02:55.586 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836491 2021-05-12T12:02:55.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:02:55.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T12:02:55.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:55.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T12:02:55.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:55.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:02:55.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T12:02:55.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:55.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T12:02:55.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T12:02:55.948 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672968 2021-05-12T12:02:55.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:56.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672968 2021-05-12T12:02:56.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:02:57.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:02:57.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:02:57.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672968 -lt 42949672968 2021-05-12T12:02:57.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:02:57.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-05-12T12:02:57.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:02:57.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:02:57.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509446 2021-05-12T12:02:57.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:02:57.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509446 2021-05-12T12:02:57.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-05-12T12:02:57.720 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509446 2021-05-12T12:02:57.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:02:58.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509447 -lt 64424509446 2021-05-12T12:02:58.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:02:58.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:58.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:58.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:02:58.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:02:58.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:02:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:02:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:02:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:02:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:02:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:02:58.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:02:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:02:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:02:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:02:59.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:02:59.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:02:59.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:02:59.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2021-05-12T12:02:59.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2021-05-12T12:02:59.768 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:02:59.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2021-05-12T12:02:59.778 INFO:tasks.workunit.client.0.smithi090.stdout:primary and soon to be divergent is 1 2021-05-12T12:02:59.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2021-05-12T12:02:59.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:03:00.121 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:03:00.121 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:02:49.895808+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:03:00.122 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:03:00.122 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:03:00.123 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:03:00.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2021-05-12T12:03:00.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:03:00.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2021-05-12T12:03:00.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2021-05-12T12:03:00.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:03:00.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2021-05-12T12:03:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2021-05-12T12:03:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T12:03:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2021-05-12T12:03:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2021-05-12T12:03:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2021-05-12T12:03:00.136 INFO:tasks.workunit.client.0.smithi090.stdout:writing initial objects 2021-05-12T12:03:00.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2021-05-12T12:03:00.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2021-05-12T12:03:00.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2021-05-12T12:03:00.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2021-05-12T12:03:00.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2021-05-12T12:03:00.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2021-05-12T12:03:00.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2021-05-12T12:03:00.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2021-05-12T12:03:00.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:00.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2021-05-12T12:03:01.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2021-05-12T12:03:01.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2021-05-12T12:03:01.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2021-05-12T12:03:01.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2021-05-12T12:03:01.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2021-05-12T12:03:01.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2021-05-12T12:03:01.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:01.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2021-05-12T12:03:02.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2021-05-12T12:03:02.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2021-05-12T12:03:02.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2021-05-12T12:03:02.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2021-05-12T12:03:02.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2021-05-12T12:03:02.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2021-05-12T12:03:02.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2021-05-12T12:03:02.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2021-05-12T12:03:02.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:02.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2021-05-12T12:03:03.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2021-05-12T12:03:03.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2021-05-12T12:03:03.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2021-05-12T12:03:03.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2021-05-12T12:03:03.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2021-05-12T12:03:03.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2021-05-12T12:03:03.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:03.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2021-05-12T12:03:04.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2021-05-12T12:03:04.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2021-05-12T12:03:04.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2021-05-12T12:03:04.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2021-05-12T12:03:04.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2021-05-12T12:03:04.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2021-05-12T12:03:04.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2021-05-12T12:03:04.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:04.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2021-05-12T12:03:05.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2021-05-12T12:03:05.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2021-05-12T12:03:05.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2021-05-12T12:03:05.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2021-05-12T12:03:05.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2021-05-12T12:03:05.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2021-05-12T12:03:05.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2021-05-12T12:03:05.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:05.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2021-05-12T12:03:06.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2021-05-12T12:03:06.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2021-05-12T12:03:06.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2021-05-12T12:03:06.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2021-05-12T12:03:06.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2021-05-12T12:03:06.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2021-05-12T12:03:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2021-05-12T12:03:06.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2021-05-12T12:03:06.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:06.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2021-05-12T12:03:07.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2021-05-12T12:03:07.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2021-05-12T12:03:07.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2021-05-12T12:03:07.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2021-05-12T12:03:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2021-05-12T12:03:07.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2021-05-12T12:03:07.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2021-05-12T12:03:07.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:07.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2021-05-12T12:03:08.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2021-05-12T12:03:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2021-05-12T12:03:08.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2021-05-12T12:03:08.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2021-05-12T12:03:08.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2021-05-12T12:03:08.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2021-05-12T12:03:08.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2021-05-12T12:03:08.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:08.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2021-05-12T12:03:09.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2021-05-12T12:03:09.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2021-05-12T12:03:09.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2021-05-12T12:03:09.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2021-05-12T12:03:09.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2021-05-12T12:03:09.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2021-05-12T12:03:09.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2021-05-12T12:03:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:09.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2021-05-12T12:03:10.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2021-05-12T12:03:10.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2021-05-12T12:03:10.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2021-05-12T12:03:10.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2021-05-12T12:03:10.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2021-05-12T12:03:10.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2021-05-12T12:03:10.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2021-05-12T12:03:10.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:10.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2021-05-12T12:03:11.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2021-05-12T12:03:11.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2021-05-12T12:03:11.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2021-05-12T12:03:11.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2021-05-12T12:03:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2021-05-12T12:03:11.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2021-05-12T12:03:11.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2021-05-12T12:03:11.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:11.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2021-05-12T12:03:12.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:12.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2021-05-12T12:03:12.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:12.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2021-05-12T12:03:12.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T12:03:12.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2021-05-12T12:03:12.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:03:12.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2021-05-12T12:03:12.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:03:12.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:03:12.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:03:12.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:03:12.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:03:12.369 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:03:12.369 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:03:12.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:03:12.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:03:12.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:03:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:03:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:03:12.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:03:12.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:03:12.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:03:12.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:03:12.783 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:03:12.783 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:03:12.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:03:12.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:12.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:03:12.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T12:03:12.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T12:03:12.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T12:03:12.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:12.921 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:03:13.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T12:03:13.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T12:03:13.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T12:03:13.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:13.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:03:13.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T12:03:13.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T12:03:13.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-05-12T12:03:13.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:13.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T12:03:13.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:13.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:03:13.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T12:03:13.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:13.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T12:03:13.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T12:03:13.189 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836496 2021-05-12T12:03:13.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:13.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T12:03:13.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:13.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T12:03:13.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:13.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:03:13.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T12:03:13.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:13.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T12:03:13.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T12:03:13.578 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672974 2021-05-12T12:03:13.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:03:13.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T12:03:13.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:13.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T12:03:13.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:13.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:03:13.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T12:03:13.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:13.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T12:03:13.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T12:03:13.961 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509452 2021-05-12T12:03:13.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:03:14.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509452 2021-05-12T12:03:14.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:03:14.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:14.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:14.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:03:14.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:14.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:14.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:14.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:14.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:14.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:14.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:15.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:03:15.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:15.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:15.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:15.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:03:15.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:03:15.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:03:15.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2021-05-12T12:03:15.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:03:15.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:03:15.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:03:15.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:03:15.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2021-05-12T12:03:15.988 INFO:tasks.workunit.client.0.smithi090.stdout:blackholing osds 0 2 2021-05-12T12:03:15.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2021-05-12T12:03:15.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:03:16.327 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:03:16.328 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:02:49.895808+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:03:16.328 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:03:16.328 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:03:16.329 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:03:16.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:16.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2021-05-12T12:03:16.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:03:16.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:03:16.340 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:03:16.341 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:03:16.341 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:03:16.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:03:16.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:03:16.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:03:16.430 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:03:16.430 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:03:16.431 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:03:16.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:16.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2021-05-12T12:03:16.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:03:16.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:03:16.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:03:16.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:03:16.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:03:16.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:03:16.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:03:16.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:03:16.529 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:03:16.530 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:03:16.530 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:03:16.537 INFO:tasks.workunit.client.0.smithi090.stdout:create new divergent objects 2021-05-12T12:03:16.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2021-05-12T12:03:16.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2021-05-12T12:03:16.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:03:16.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:03:16.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2021-05-12T12:03:16.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2021-05-12T12:03:16.540 INFO:tasks.workunit.client.0.smithi090.stdout:writing divergent objects 2021-05-12T12:03:16.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2021-05-12T12:03:16.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2021-05-12T12:03:16.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:03:16.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:03:16.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T12:03:16.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2021-05-12T12:03:16.543 INFO:tasks.workunit.client.0.smithi090.stdout:remove divergent objects 2021-05-12T12:03:16.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T12:03:16.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2021-05-12T12:03:16.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:03:16.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2021-05-12T12:03:16.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2021-05-12T12:03:16.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:03:16.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2021-05-12T12:03:16.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2021-05-12T12:03:16.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2021-05-12T12:03:16.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2021-05-12T12:03:16.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2021-05-12T12:03:26.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2021-05-12T12:03:26.550 INFO:tasks.workunit.client.0.smithi090.stderr:rados: no process found 2021-05-12T12:03:26.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2021-05-12T12:03:26.551 INFO:tasks.workunit.client.0.smithi090.stdout:killing all the osds 2021-05-12T12:03:26.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:03:26.900 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:03:26.900 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2021-05-12T12:02:49.895808+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:03:26.900 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:03:26.900 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:03:26.901 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:03:26.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:03:26.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:03:26.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:03:26.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:03:26.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:03:26.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:03:27.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:03:27.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:03:27.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2021-05-12T12:03:27.517 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:03:27.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:03:27.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2021-05-12T12:03:28.028 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:03:28.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T12:03:28.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2021-05-12T12:03:28.538 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:03:28.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:28.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2021-05-12T12:03:29.071 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already out. 2021-05-12T12:03:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:29.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2021-05-12T12:03:29.582 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already out. 2021-05-12T12:03:29.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2021-05-12T12:03:29.591 INFO:tasks.workunit.client.0.smithi090.stdout:bringing up non_divergent 0 2 2021-05-12T12:03:29.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:03:29.929 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:03:29.930 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2021-05-12T12:02:49.895808+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:03:29.930 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:03:29.930 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:03:29.931 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:03:29.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:29.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2021-05-12T12:03:29.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:03:29.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:03:29.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:03:29.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:03:29.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:03:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:03:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:03:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:03:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:03:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:03:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:03:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:03:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:03:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:03:29.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:03:29.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:03:29.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:03:29.947 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:03:29.947 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:03:29.948 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:03:29.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:03:29.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:03:29.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:03:29.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:03:29.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:03:29.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:03:29.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:03:29.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:29.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:03:29.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:03:29.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:03:29.950 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:03:29.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:03:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:29.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:03:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:03:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:03:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:03:29.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:03:29.971 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:29.969+0000 7f011a7750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:29.977 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:29.974+0000 7f011a7750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:29.978 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:29.976+0000 7f011a7750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:30.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:03:30.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:03:30.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:03:30.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:03:30.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:03:30.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:30.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:03:30.313 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:03:30.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:30.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:03:30.493 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:30.491+0000 7f011a7750c0 -1 Falling back to public interface 2021-05-12T12:03:30.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:31.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:31.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:31.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:03:31.696 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:03:31.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:31.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:03:32.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:33.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:33.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:33.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:03:33.089 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:03:33.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:33.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:03:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:33.116+0000 7f011a7750c0 -1 osd.0 28 log_to_monitors {default=true} 2021-05-12T12:03:33.468 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/49363,v1:127.0.0.1:6803/49363] [v2:127.0.0.1:6804/49363,v1:127.0.0.1:6805/49363] exists,up f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:03:33.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:03:33.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:03:33.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:03:33.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:33.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-05-12T12:03:33.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:03:33.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:03:33.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:03:33.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:03:33.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:03:33.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:03:33.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:03:33.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:03:33.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:03:33.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:03:33.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:03:33.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:03:33.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:03:33.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:03:33.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:03:33.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:03:33.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:03:33.478 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:03:33.478 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:03:33.478 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:03:33.478 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:03:33.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:03:33.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:03:33.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:03:33.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:03:33.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:03:33.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:03:33.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:33.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:03:33.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:03:33.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:03:33.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:03:33.481 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:03:33.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:03:33.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:03:33.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:03:33.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:03:33.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:03:33.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:03:33.500 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:33.498+0000 7f30069150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:33.539 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:33.537+0000 7f30069150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:33.540 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:33.538+0000 7f30069150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:03:33.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:03:33.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:03:33.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:03:33.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:03:33.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:03:33.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:33.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:03:33.885 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:03:33.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:33.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:34.056 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:34.053+0000 7f30069150c0 -1 Falling back to public interface 2021-05-12T12:03:34.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:03:35.253 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:03:35.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:35.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:35.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:03:36.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:03:36.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:03:36.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:03:36.651 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:03:36.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:03:36.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:03:36.778 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:03:36.776+0000 7f30069150c0 -1 osd.2 28 log_to_monitors {default=true} 2021-05-12T12:03:37.051 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/49920,v1:127.0.0.1:6811/49920] [v2:127.0.0.1:6812/49920,v1:127.0.0.1:6813/49920] exists,up c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:03:37.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:03:37.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:03:37.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:03:37.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:37.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2021-05-12T12:03:37.616 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already in. 2021-05-12T12:03:37.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:03:37.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2021-05-12T12:03:38.275 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already in. 2021-05-12T12:03:38.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:03:38.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2021-05-12T12:03:38.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:03:38.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:03:38.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:03:38.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:03:38.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:03:38.287 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:03:38.287 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:03:38.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:03:38.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:03:38.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:03:38.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:03:38.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:03:38.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:03:38.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:03:38.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:03:38.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:03:38.707 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:03:38.707 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:03:38.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:03:38.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:38.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:03:38.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=150323855364 2021-05-12T12:03:38.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 150323855364 2021-05-12T12:03:38.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855364' 2021-05-12T12:03:38.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:38.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:03:38.938 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:03:38.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:03:38.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:03:38.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:03:38.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:03:38.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:03:39.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=158913789955 2021-05-12T12:03:39.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 158913789955 2021-05-12T12:03:39.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2021-05-12T12:03:39.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:39.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855364 2021-05-12T12:03:39.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:39.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:03:39.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-150323855364 2021-05-12T12:03:39.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:39.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=150323855364 2021-05-12T12:03:39.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2021-05-12T12:03:39.075 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 150323855364 2021-05-12T12:03:39.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:39.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 150323855364 2021-05-12T12:03:39.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:03:40.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:03:40.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:03:40.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855364 -lt 150323855364 2021-05-12T12:03:40.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:03:40.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-158913789955 2021-05-12T12:03:40.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:03:40.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:03:40.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-158913789955 2021-05-12T12:03:40.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:03:40.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=158913789955 2021-05-12T12:03:40.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2021-05-12T12:03:40.811 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 158913789955 2021-05-12T12:03:40.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:03:41.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 158913789955 -lt 158913789955 2021-05-12T12:03:41.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:03:41.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:41.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:41.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:03:41.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:41.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:41.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:41.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:41.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:41.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:41.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:41.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:41.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:41.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:41.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:42.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:42.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:03:42.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:03:42.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:03:42.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:42.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:03:42.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:42.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:42.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:42.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:42.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:42.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:42.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:42.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:42.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:42.921 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:42.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:42.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:43.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:43.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:43.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:43.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:43.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:43.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:03:43.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:43.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:03:44.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:44.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:44.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:44.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:44.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:44.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:44.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:44.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:44.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:44.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:44.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:44.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:44.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:44.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:44.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:44.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:44.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:44.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:44.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:44.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:44.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:45.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:45.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:45.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:03:45.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:45.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:03:45.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:45.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:45.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:45.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:46.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:46.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:46.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:46.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:46.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:46.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:46.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:46.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:46.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:46.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:46.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:46.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:46.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:47.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:03:47.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:47.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:03:47.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:47.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:47.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:47.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:47.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:47.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:47.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:47.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:48.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:48.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:48.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:48.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:48.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:48.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:48.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:48.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:48.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:48.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:48.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:48.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:48.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:49.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:49.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:49.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:03:49.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:49.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:03:50.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:50.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:50.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:50.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:50.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:50.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:50.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:50.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:51.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:51.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:51.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:51.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:51.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:51.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:51.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:51.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:51.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:51.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:51.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:51.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:51.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:52.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:52.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:52.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:03:52.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:52.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:03:55.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:03:55.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:03:55.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:03:55.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:03:55.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:03:55.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:03:55.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:03:55.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:03:55.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:03:55.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:03:55.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:03:55.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:03:56.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:03:56.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:03:56.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:03:56.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:03:56.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:03:56.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:03:56.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:03:56.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:03:56.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:03:56.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:03:56.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:03:56.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:03:56.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:03:56.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:04:03.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:03.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:03.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:03.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:03.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:03.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:03.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:03.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:03.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:03.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:03.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:03.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:03.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:03.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:03.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:03.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:03.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:03.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:03.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:03.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:03.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:04.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:04.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:04.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:04:04.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:04.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:04:11.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:11.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:11.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:11.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:11.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:11.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:11.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:11.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:12.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:12.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:12.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:12.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:12.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:12.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:12.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:12.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:12.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:12.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:12.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:12.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:12.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:13.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:13.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:13.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:04:13.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:04:13.501 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:04:13.501 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "5599757c-0a32-4d62-a605-2660bacac11c", 2021-05-12T12:04:13.501 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:13.502 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:04:13.502 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:04:13.498152+0000", 2021-05-12T12:04:13.502 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:04:13.502 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:04:13.503 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:04:13.503 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:04:13.503 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:04:13.503 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:04:13.504 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:13.504 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:04:13.504 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:13.504 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.504 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:13.505 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.505 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:04:13.505 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.505 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.506 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:13.506 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.506 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:04:13.506 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:13.506 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:13.507 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:04:13.507 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:04:13.507 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.507 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:04:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:04:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:13.509 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:04:13.509 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:13.509 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.509 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:13.509 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.510 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:04:13.510 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.510 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.510 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:13.511 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.511 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:04:13.511 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:13.511 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:13.511 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:04:13.512 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:13.512 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.512 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:13.512 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.513 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:04:13.513 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.513 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.513 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:13.513 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.514 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:04:13.514 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:13.514 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:13.514 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:04:13.515 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:13.515 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.515 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:13.515 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.516 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:04:13.516 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.516 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.516 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:04:13.516 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.517 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.517 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:13.517 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.517 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.518 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:04:13.518 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.518 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:04:13.518 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:04:13.518 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:04:13.519 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:04:13.519 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:04:13.519 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:02:17.591619Z", 2021-05-12T12:04:13.519 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:02:17.591619Z", 2021-05-12T12:04:13.520 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:04:13.520 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:04:13.520 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:04:13.520 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:04:13.520 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:04:13.521 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:04:13.521 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:04:13.521 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:04:13.521 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:04:13.521 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:04:13.522 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:04:13.522 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:04:13.522 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:04:13.522 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:04:13.523 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:04:13.523 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:04:13.523 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.524 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:04:13.524 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.524 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:04:13.524 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.524 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:04:13.525 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:04:13.525 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:13.525 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.525 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.526 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.526 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:04:13.526 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:04:13.526 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.526 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.527 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.527 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:04:13.527 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:04:13.527 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:04:13.528 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:04:13.528 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:04:13.528 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.528 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.528 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.529 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:04:13.529 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:04:13.529 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.529 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:04:13.529 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 41, 2021-05-12T12:04:13.530 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:04:13.530 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:02:18.428606+0000", 2021-05-12T12:04:13.530 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:03:38.377583+0000", 2021-05-12T12:04:13.530 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:03:36.967722+0000", 2021-05-12T12:04:13.530 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:03:37.937081+0000", 2021-05-12T12:04:13.531 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:04:13.531 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:04:13.531 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:04:13.531 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:04:13.532 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:04:13.532 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:04:13.532 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:04:13.532 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:04:13.532 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:04:13.533 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:04:13.533 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.533 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 9, 2021-05-12T12:04:13.533 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:04:13.533 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:04:13.534 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:04:13.534 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:04:13.534 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:04:13.534 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:04:13.535 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:04:13.535 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:04:13.535 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:04:13.535 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:04:13.535 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.536 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:04:13.536 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:04:13.536 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:02:48.804476+0000", 2021-05-12T12:04:13.536 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 1, 2021-05-12T12:04:13.536 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool", 2021-05-12T12:04:13.537 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:04:13.537 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:04:13.537 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:04:13.537 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 0, 2021-05-12T12:04:13.537 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:04:13.538 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:04:13.538 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:04:13.538 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:04:13.538 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:04:13.538 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:04:13.539 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 1, 2021-05-12T12:04:13.539 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:04:13.539 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 1, 2021-05-12T12:04:13.539 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 1, 2021-05-12T12:04:13.540 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 1, 2021-05-12T12:04:13.540 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:04:13.540 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:04:13.540 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:04:13.540 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:04:13.541 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:04:13.541 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:04:13.541 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:04:13.541 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.541 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "28", 2021-05-12T12:04:13.542 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:04:13.542 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:04:13.542 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:04:13.542 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:04:13.543 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:04:13.543 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 0, 2021-05-12T12:04:13.543 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 0, 2021-05-12T12:04:13.543 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [], 2021-05-12T12:04:13.543 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:04:13.544 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:04:13.544 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:04:13.544 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:04:13.544 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:04:13.544 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:04:13.545 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:04:13.545 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:04:13.545 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:04:13.545 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:04:13.546 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:04:13.546 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:04:13.546 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:04:13.546 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:04:13.546 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:04:13.547 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "", 2021-05-12T12:04:13.547 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:04:13.547 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:04:13.547 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.548 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:04:13.548 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:04:13.548 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:04:13.548 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:04:13.548 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:04:13.549 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:04:13.549 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:04:13.549 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:04:13.549 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 0, 2021-05-12T12:04:13.549 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:04:13.550 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:04:13.550 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:04:13.550 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:04:13.550 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.550 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.551 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:04:13.551 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.551 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:13.551 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "f2cb872d-ef34-4fe5-a813-b277592bb9a2", 2021-05-12T12:04:13.551 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:04:13.552 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:13.552 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:13.552 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:13.552 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:04:13.553 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:04:13.553 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 35, 2021-05-12T12:04:13.553 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 39, 2021-05-12T12:04:13.553 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 29, 2021-05-12T12:04:13.553 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:13.554 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:13.554 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.554 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.554 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.554 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:04:13.555 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.555 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.555 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.555 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.555 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:04:13.556 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.556 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.556 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.556 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.557 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:13.557 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.557 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.557 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.557 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:04:13.558 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.558 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.558 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.558 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.558 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:04:13.559 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.559 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.559 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.559 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.560 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:13.560 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.560 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.560 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.560 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:04:13.561 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.561 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.561 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.561 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.561 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:04:13.562 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.562 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.562 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.562 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.563 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:13.563 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.563 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.563 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.563 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:04:13.564 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.564 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.564 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.564 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.564 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:04:13.565 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:13.565 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.565 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.565 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.565 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/49363", 2021-05-12T12:04:13.566 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/49363", 2021-05-12T12:04:13.566 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/49363", 2021-05-12T12:04:13.566 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/49363", 2021-05-12T12:04:13.566 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:13.567 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:04:13.567 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:04:13.567 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.567 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.567 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.568 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:13.568 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "9f1bd5b2-c83e-463c-b023-d48ebd49764a", 2021-05-12T12:04:13.568 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:04:13.568 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:13.568 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:13.569 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:13.569 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:04:13.569 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:04:13.569 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 10, 2021-05-12T12:04:13.569 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 22, 2021-05-12T12:04:13.570 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 30, 2021-05-12T12:04:13.570 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:13.570 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:13.570 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.570 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.571 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.571 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:13.571 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.571 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.571 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.572 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.572 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:13.572 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.572 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.573 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.573 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.573 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:13.573 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.573 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.574 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.574 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:13.574 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.574 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.574 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.575 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.575 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:13.575 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.575 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.575 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.576 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.576 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:13.576 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.576 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.577 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.577 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:13.577 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.577 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.577 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.578 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.578 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:13.578 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.578 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.578 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.579 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.579 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:13.579 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.579 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.579 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.580 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:13.580 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.580 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.580 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.580 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.581 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:13.581 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:13.581 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.581 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.582 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.582 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/43911", 2021-05-12T12:04:13.582 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/43911", 2021-05-12T12:04:13.583 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43911", 2021-05-12T12:04:13.583 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43911", 2021-05-12T12:04:13.583 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:13.583 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:04:13.583 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.584 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.584 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.584 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:13.584 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0", 2021-05-12T12:04:13.585 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:04:13.585 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:13.585 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:13.585 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:13.586 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 15, 2021-05-12T12:04:13.586 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:04:13.586 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 37, 2021-05-12T12:04:13.586 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:04:13.586 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 31, 2021-05-12T12:04:13.587 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:13.587 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:13.587 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.587 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.588 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.588 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:13.588 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.588 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.589 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.589 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.589 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:13.589 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.589 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.590 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.590 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.590 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:13.590 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.591 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.591 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.591 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:13.591 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.592 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.592 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.592 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.592 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:13.593 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.593 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.593 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.593 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.593 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:13.594 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.594 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.594 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.594 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:13.595 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.595 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.595 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.595 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.595 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:13.596 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.596 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.596 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.596 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.597 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:13.597 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:13.597 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.597 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:13.598 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:13.598 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.598 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.598 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.598 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:13.599 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:13.599 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:13.599 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.599 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.600 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.600 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/49920", 2021-05-12T12:04:13.600 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/49920", 2021-05-12T12:04:13.600 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/49920", 2021-05-12T12:04:13.600 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/49920", 2021-05-12T12:04:13.601 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:13.601 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:04:13.601 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:04:13.601 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.601 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.602 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.602 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:04:13.602 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.602 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:13.603 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:27.325634+0000", 2021-05-12T12:04:13.603 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:13.603 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:13.603 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:13.604 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:13.604 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:28.169537+0000", 2021-05-12T12:04:13.604 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:13.604 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.604 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.605 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:13.605 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:27.838193+0000", 2021-05-12T12:04:13.605 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:13.605 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:13.605 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:13.606 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:13.606 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:37.353029+0000", 2021-05-12T12:04:13.606 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:13.606 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.607 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.607 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:13.607 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:28.348696+0000", 2021-05-12T12:04:13.607 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:13.607 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:13.608 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:13.608 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:13.608 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:46.582993+0000", 2021-05-12T12:04:13.608 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:13.609 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.609 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.609 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:04:13.609 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:04:13.609 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:04:13.610 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:04:13.610 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:04:13.610 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:04:13.610 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:04:13.611 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:04:13.611 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:04:13.611 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:04:13.611 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:04:13.611 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.612 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.612 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:04:13.612 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:04:13.612 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:04:13.613 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:04:13.613 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:04:13.613 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:04:13.613 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:04:13.614 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:04:13.614 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:04:13.614 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:04:13.614 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:04:13.614 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.615 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.615 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:04:13.615 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.615 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:13.616 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:13.616 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/49363,v1:127.0.0.1:6805/49363]", 2021-05-12T12:04:13.616 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:13.616 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:13.617 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:13.617 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:13.617 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:13.617 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:13.617 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:13.618 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:13.618 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:13.618 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:04:13.618 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:13.619 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:13.619 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:13.619 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:13.619 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:13.620 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:13.620 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:13.620 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:13.620 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:13.620 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:13.621 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:13.621 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:13.621 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:13.621 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:13.622 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:13.622 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/49363,v1:127.0.0.1:6803/49363]", 2021-05-12T12:04:13.622 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:13.622 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/49363,v1:127.0.0.1:6809/49363]", 2021-05-12T12:04:13.622 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/49363,v1:127.0.0.1:6807/49363]", 2021-05-12T12:04:13.623 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:13.623 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:13.623 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:13.623 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:13.624 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:13.624 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:13.624 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:13.624 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:13.625 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:13.625 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:04:13.625 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:13.625 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:13.625 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:13.626 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.626 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.626 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:13.626 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:13.626 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/43911,v1:127.0.0.1:6813/43911]", 2021-05-12T12:04:13.627 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:13.627 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:13.627 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:13.627 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:13.628 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:13.628 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:13.628 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:13.628 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:13.628 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:13.629 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:04:13.629 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:13.629 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:13.629 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:13.630 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:13.630 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:13.630 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:13.630 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:13.630 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:13.631 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:13.631 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:13.631 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:13.631 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:13.631 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:13.632 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:13.632 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:13.632 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/43911,v1:127.0.0.1:6811/43911]", 2021-05-12T12:04:13.632 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:13.633 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43911,v1:127.0.0.1:6817/43911]", 2021-05-12T12:04:13.633 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43911,v1:127.0.0.1:6815/43911]", 2021-05-12T12:04:13.633 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:13.633 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:13.633 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:13.634 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:13.634 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:13.634 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:13.634 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:13.634 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:13.635 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:13.635 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:04:13.635 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:13.635 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:13.636 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:13.636 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.636 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.636 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:13.636 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:13.637 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/49920,v1:127.0.0.1:6813/49920]", 2021-05-12T12:04:13.637 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:13.637 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:13.637 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:13.638 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:13.638 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:13.638 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:13.638 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:13.638 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:13.639 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:13.639 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:04:13.639 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:13.639 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:13.640 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:13.640 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:13.640 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:13.640 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:13.641 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:13.641 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:13.641 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:13.641 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:13.641 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:13.642 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:13.642 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:13.642 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:13.642 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:13.642 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/49920,v1:127.0.0.1:6811/49920]", 2021-05-12T12:04:13.643 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:13.643 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/49920,v1:127.0.0.1:6817/49920]", 2021-05-12T12:04:13.643 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/49920,v1:127.0.0.1:6815/49920]", 2021-05-12T12:04:13.643 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:13.643 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:13.644 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:13.644 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:13.644 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:13.644 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:13.645 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:13.645 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:13.645 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:13.645 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:04:13.645 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:13.646 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:13.646 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:13.646 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.646 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.646 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:04:13.647 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:04:13.647 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:04:13.647 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:04:13.647 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.648 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:04:13.648 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.648 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:13.648 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 35 2021-05-12T12:04:13.648 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.649 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.649 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:13.649 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 10 2021-05-12T12:04:13.649 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.649 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.650 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:13.650 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 38 2021-05-12T12:04:13.650 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.650 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.651 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.651 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:04:13.651 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 41, 2021-05-12T12:04:13.651 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:04:13.651 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:04:13.652 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.652 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:13.652 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:04:13.652 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:13.652 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.653 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.653 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:13.653 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:04:13.653 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:13.653 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.654 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.654 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:13.654 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:04:13.654 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:13.655 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.655 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.655 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:04:13.655 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.655 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:04:13.656 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:04:13.656 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.656 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.656 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:13.656 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:04:13.657 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.657 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.657 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:04:13.657 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:04:13.658 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.658 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.658 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:04:13.658 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:04:13.658 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.659 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.659 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:04:13.659 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:04:13.659 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.660 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.660 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:04:13.660 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:04:13.660 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.660 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.661 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:04:13.661 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:04:13.661 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.661 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.661 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:04:13.662 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:04:13.662 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.662 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.662 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:04:13.663 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:04:13.663 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.663 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.663 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:04:13.663 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:04:13.664 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.664 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.664 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:04:13.664 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:04:13.665 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.665 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.665 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:13.665 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:04:13.665 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.666 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.666 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:04:13.666 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.666 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:04:13.666 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:04:13.667 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:13.667 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:04:13.667 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.667 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:13.668 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:13.668 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:13.668 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.668 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:04:13.669 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.669 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:13.669 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.669 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.669 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.670 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.670 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:04:13.670 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:04:13.670 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:13.670 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:04:13.671 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.671 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:13.671 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:13.671 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:13.672 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.672 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:04:13.672 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.672 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:13.673 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.673 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.673 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.673 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.673 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:04:13.674 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:04:13.674 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:13.674 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:04:13.674 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.675 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:13.675 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:13.675 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:13.675 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.675 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:13.676 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.676 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:13.676 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.676 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.677 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:13.677 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.677 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:04:13.677 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.677 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.678 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:13.678 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.678 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:04:13.678 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.678 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.679 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.679 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.679 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:04:13.679 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:04:13.680 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:13.680 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:04:13.680 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:13.680 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:13.680 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:13.681 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:13.681 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.681 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:13.681 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.682 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:13.682 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.682 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.682 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:13.682 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.683 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:04:13.683 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.683 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.683 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:13.684 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:13.684 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:04:13.684 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.684 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.684 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.685 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.685 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:04:13.685 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.685 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:04:13.686 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:04:13.686 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:04:13.686 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:04:13.686 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:04:13.686 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:04:13.687 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:04:13.687 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.687 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:04:13.687 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:04:13.687 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:04:13.688 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.688 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.688 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:04:13.688 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:04:13.689 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:04:13.689 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.689 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.689 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:04:13.689 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.690 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:13.690 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.690 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.690 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:04:13.691 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:04:13.691 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:04:13.691 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:04:13.691 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:04:13.691 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:04:13.692 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:04:13.692 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:04:13.692 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:04:13.692 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:04:13.693 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:04:13.693 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:04:13.693 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:04:13.693 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:04:13.694 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:04:13.694 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 0, 2021-05-12T12:04:13.694 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:04:13.694 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:04:13.695 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:04:13.695 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:04:13.695 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:04:13.695 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.695 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:04:13.696 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.696 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:04:13.696 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:04:13.696 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:04:13.697 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:04:13.697 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:04:13.697 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:04:13.697 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:04:13.698 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:04:13.698 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:04:13.698 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:04:13.698 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:04:13.698 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:04:13.699 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:04:13.699 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:04:13.699 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:04:13.699 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:04:13.700 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.700 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.700 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:04:13.700 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:04:13.700 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:04:13.701 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.701 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:04:13.701 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:04:13.701 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.701 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:04:13.702 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:04:13.702 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:04:13.702 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:04:13.702 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:04:13.703 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:04:13.703 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.703 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:04:13.703 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 1, 2021-05-12T12:04:13.703 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:04:13.704 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:04:13.704 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:04:13.704 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:04:13.704 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:04:13.705 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:04:13.705 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:04:13.705 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:04:13.705 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:13.705 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:04:13.706 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:13.706 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:13.706 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:13.706 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:13.707 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:04:13.707 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:04:13.707 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:04:13.707 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:04:13.707 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:04:13.708 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:13.708 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:13.708 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:13.708 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:13.708 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:13.709 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:13.709 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:04:13.709 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:13.709 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:13.710 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:04:13.710 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:04:13.710 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:04:13.710 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:04:13.710 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:04:13.711 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:13.711 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:13.711 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:13.711 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:13.712 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:13.712 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:13.712 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:13.712 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:13.712 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:13.713 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:04:13.713 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:04:13.713 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.713 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:04:13.713 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:04:13.714 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:04:13.714 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:13.714 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:04:13.714 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:04:13.715 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:13.715 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:13.715 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:13.715 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:04:13.716 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:04:13.716 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.716 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 16, 2021-05-12T12:04:13.716 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 16, 2021-05-12T12:04:13.716 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:04:13.717 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:04:13.717 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:04:13.717 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.717 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:04:13.717 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:04:13.718 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:04:13.718 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:04:13.718 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:04:13.718 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:13.719 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:13.719 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:04:13.719 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18448, 2021-05-12T12:04:13.719 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2096, 2021-05-12T12:04:13.719 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:04:13.720 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:13.720 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554352, 2021-05-12T12:04:13.720 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:04:13.720 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:04:13.721 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103656448, 2021-05-12T12:04:13.721 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:13.721 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2146304, 2021-05-12T12:04:13.721 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 415020, 2021-05-12T12:04:13.722 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:13.722 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:13.722 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:13.722 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:04:13.723 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:04:13.723 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.723 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:04:13.723 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:13.723 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:13.724 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:13.724 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:04:13.724 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:04:13.724 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:04:13.724 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.725 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:04:13.725 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:13.725 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:13.725 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:13.726 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:04:13.726 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.726 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:04:13.726 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:04:13.726 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.727 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:04:13.727 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:04:13.727 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:04:13.727 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:04:13.728 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:04:13.728 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:04:13.728 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:13.728 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:13.728 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:04:13.729 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18448, 2021-05-12T12:04:13.729 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2096, 2021-05-12T12:04:13.729 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:04:13.729 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:13.729 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554352, 2021-05-12T12:04:13.730 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:04:13.730 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:04:13.730 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103656448, 2021-05-12T12:04:13.730 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:13.731 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2146304, 2021-05-12T12:04:13.731 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 415020, 2021-05-12T12:04:13.731 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:13.731 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:13.731 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:13.732 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:04:13.732 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:04:13.732 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.732 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:04:13.733 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:13.733 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:13.733 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:13.733 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:04:13.734 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:04:13.734 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:04:13.734 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.734 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:04:13.734 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:13.735 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:13.735 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:13.735 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:04:13.735 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.735 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:04:13.736 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:04:13.736 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.736 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.736 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:04:13.736 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.737 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:04:13.737 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:04:13.737 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:04:13.737 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:04:13.737 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:04:13.738 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:04:13.738 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:04:13.738 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:13.738 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:04:13.739 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:13.739 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:13.739 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:13.739 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:13.740 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:04:13.740 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:04:13.740 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:04:13.740 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:04:13.740 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:04:13.741 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:13.741 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:13.741 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:13.741 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:13.742 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:13.742 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:13.742 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:04:13.742 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:13.742 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:13.743 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:04:13.743 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:04:13.743 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:04:13.743 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:04:13.743 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:04:13.744 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:13.744 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:13.744 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:13.744 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:13.744 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:13.745 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:13.745 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:13.745 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:13.745 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:13.745 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:04:13.746 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:04:13.746 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.746 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:04:13.746 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:04:13.747 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:04:13.747 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:13.747 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1220608, 2021-05-12T12:04:13.747 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 143228, 2021-05-12T12:04:13.747 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:13.748 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:13.748 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:13.748 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:04:13.748 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:04:13.748 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.749 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 16, 2021-05-12T12:04:13.749 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 16, 2021-05-12T12:04:13.749 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:04:13.749 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:04:13.750 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:04:13.750 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.750 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.750 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:04:13.750 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.751 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:13.751 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 150323855371 2021-05-12T12:04:13.751 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.751 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.751 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:13.752 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 42949672976 2021-05-12T12:04:13.752 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.752 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.752 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:13.753 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 158913789962 2021-05-12T12:04:13.753 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.753 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.753 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:04:13.753 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.754 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:04:13.754 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 1 2021-05-12T12:04:13.754 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.754 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.754 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:04:13.755 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.755 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:13.755 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:04:13.755 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:04:13.755 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:13.756 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.756 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.756 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:13.756 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:04:13.756 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:04:13.757 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:13.757 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.757 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.757 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:13.758 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 1, 2021-05-12T12:04:13.758 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:04:13.758 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:13.758 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.758 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.759 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:04:13.759 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:13.759 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:04:13.759 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:04:13.760 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.760 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:13.760 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:04:13.760 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 4, 2021-05-12T12:04:13.760 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:03:39.382006+0000", 2021-05-12T12:04:13.761 INFO:tasks.workunit.client.0.smithi090.stdout: "services": { 2021-05-12T12:04:13.761 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": { 2021-05-12T12:04:13.761 INFO:tasks.workunit.client.0.smithi090.stdout: "daemons": { 2021-05-12T12:04:13.761 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": "", 2021-05-12T12:04:13.761 INFO:tasks.workunit.client.0.smithi090.stdout: "0": { 2021-05-12T12:04:13.762 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:04:13.762 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:04:13.762 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:04:13.762 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:04:13.763 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:04:13.763 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:04:13.763 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.763 INFO:tasks.workunit.client.0.smithi090.stdout: "2": { 2021-05-12T12:04:13.763 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:04:13.764 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:04:13.764 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:04:13.764 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:04:13.764 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:04:13.764 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:04:13.765 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.765 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.765 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.765 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.766 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:13.766 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:04:13.766 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:04:13.766 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 361, 2021-05-12T12:04:13.766 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:04:13.767 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:04:13.767 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:13.767 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:04:13.768 INFO:tasks.workunit.client.0.smithi090.stderr:report 2856303984 2021-05-12T12:04:13.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:04:13.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2021-05-12T12:04:13.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2021-05-12T12:04:13.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2021-05-12T12:04:13.769 INFO:tasks.workunit.client.0.smithi090.stdout:writing non-divergent object existing_4 2021-05-12T12:04:13.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:13.851 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:13.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 28'100 41:122 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:13.852 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:13.852 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:13.853 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:04:13.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:04:13.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2021-05-12T12:04:13.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:04:13.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:04:13.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:04:13.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:04:13.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:04:13.960 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:04:13.960 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:04:13.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:04:13.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:04:13.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:04:14.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:04:14.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:04:14.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:04:14.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:04:14.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:04:14.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:04:14.378 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:04:14.378 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:04:14.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:04:14.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:04:14.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:04:14.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=150323855373 2021-05-12T12:04:14.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 150323855373 2021-05-12T12:04:14.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855373' 2021-05-12T12:04:14.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:04:14.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:04:14.608 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:04:14.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:04:14.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:04:14.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:04:14.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:04:14.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:04:14.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=158913789964 2021-05-12T12:04:14.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 158913789964 2021-05-12T12:04:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-150323855373 2-158913789964' 2021-05-12T12:04:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:04:14.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855373 2021-05-12T12:04:14.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:04:14.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:04:14.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-150323855373 2021-05-12T12:04:14.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:04:14.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=150323855373 2021-05-12T12:04:14.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2021-05-12T12:04:14.746 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 150323855373 2021-05-12T12:04:14.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:04:15.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855371 -lt 150323855373 2021-05-12T12:04:15.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:04:16.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:04:16.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:04:16.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855373 -lt 150323855373 2021-05-12T12:04:16.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:04:16.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-158913789964 2021-05-12T12:04:16.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:04:16.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:04:16.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-158913789964 2021-05-12T12:04:16.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:04:16.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=158913789964 2021-05-12T12:04:16.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 158913789964' 2021-05-12T12:04:16.484 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 158913789964 2021-05-12T12:04:16.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:04:16.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 158913789964 -lt 158913789964 2021-05-12T12:04:16.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:04:16.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:16.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:17.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:04:17.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:17.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:17.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:17.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:17.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:17.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:17.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:17.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:17.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:17.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:17.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:18.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:04:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:04:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:04:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:18.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:04:18.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:18.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:18.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:18.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:18.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:18.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:18.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:18.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:18.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:18.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:18.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:18.612 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:19.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:19.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:19.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:19.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:19.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:19.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:19.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:19.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:19.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:19.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:19.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:19.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:04:19.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:19.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:04:19.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:19.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:19.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:19.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:19.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:19.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:19.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:19.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:20.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:20.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:20.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:20.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:20.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:20.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:20.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:20.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:20.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:20.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:20.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:20.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:20.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:21.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:21.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:21.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:04:21.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:21.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:04:21.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:21.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:21.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:21.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:21.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:21.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:21.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:21.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:21.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:21.824 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:21.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:21.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:22.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:22.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:22.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:22.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:22.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:22.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:22.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:22.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:22.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:22.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:22.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:22.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:04:22.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:22.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:04:23.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:23.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:23.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:23.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:23.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:23.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:23.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:23.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:23.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:23.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:23.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:23.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:24.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:24.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:24.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:24.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:24.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:24.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:24.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:24.408 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:24.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:24.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:24.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:24.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:04:24.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:24.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:04:26.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:26.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:26.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:26.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:26.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:26.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:26.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:26.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:26.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:26.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:26.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:26.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:27.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:27.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:27.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:27.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:27.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:27.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:27.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:27.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:27.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:27.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:27.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:27.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:04:27.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:27.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:04:30.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:30.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:30.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:30.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:31.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:31.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:31.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:31.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:31.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:31.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:31.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:31.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:31.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:31.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:31.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:31.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:31.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:32.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:32.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:32.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:04:32.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:32.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:04:38.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:38.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:38.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:38.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:38.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:38.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:38.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:38.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:39.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:39.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:39.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:39.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:39.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:39.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:39.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:39.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:39.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:39.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:39.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:39.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:39.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:40.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:40.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:40.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:04:40.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:04:40.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:04:47.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:04:47.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:04:47.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:04:47.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:04:47.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:04:47.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:04:47.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:04:47.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:04:47.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:04:47.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:04:47.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:04:47.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:04:48.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:04:48.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:04:48.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:04:48.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:04:48.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:04:48.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:04:48.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:04:48.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:04:48.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:04:48.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:04:48.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:04:48.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:04:48.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:04:49.168 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:04:49.168 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "5599757c-0a32-4d62-a605-2660bacac11c", 2021-05-12T12:04:49.169 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:49.169 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:04:49.169 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:04:49.165451+0000", 2021-05-12T12:04:49.169 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:04:49.170 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:04:49.170 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:04:49.170 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:04:49.170 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:04:49.171 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:04:49.171 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:49.171 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:04:49.171 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:49.172 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.172 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:49.172 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.172 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:04:49.172 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.173 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.173 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:49.173 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.174 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:04:49.174 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:49.174 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:49.174 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:04:49.174 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:04:49.175 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.175 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:04:49.175 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:49.175 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.176 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:04:49.176 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:49.176 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:49.176 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:04:49.176 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:49.177 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.177 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:49.177 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.177 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:04:49.178 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.178 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.178 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:49.178 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.178 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:04:49.179 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:49.179 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:49.179 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2021-05-12T12:04:49.179 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 2 2021-05-12T12:04:49.180 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.180 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:49.180 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.180 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.0 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:04:49.181 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.181 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.181 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:49.181 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.181 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:04:49.182 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:04:49.182 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:04:49.182 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:04:49.182 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:04:49.183 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.183 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:04:49.183 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.183 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:04:49.183 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.184 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.184 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:04:49.184 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.184 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.185 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:04:49.185 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.185 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.185 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:04:49.185 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.186 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:04:49.186 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:04:49.186 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:04:49.186 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:04:49.187 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:04:49.187 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:02:17.591619Z", 2021-05-12T12:04:49.187 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:02:17.591619Z", 2021-05-12T12:04:49.187 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:04:49.187 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:04:49.188 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:04:49.188 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:04:49.188 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:04:49.188 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:04:49.188 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:04:49.189 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:04:49.189 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:04:49.189 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:04:49.189 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:04:49.190 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:04:49.190 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:04:49.190 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:04:49.190 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:04:49.191 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:04:49.191 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.191 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:04:49.191 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.191 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:04:49.192 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.192 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:04:49.192 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:04:49.192 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:49.193 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.193 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.193 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.193 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:04:49.193 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:04:49.194 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.194 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.194 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.194 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:04:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:04:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:04:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:04:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:04:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.196 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.196 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.196 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:04:49.196 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:04:49.196 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.197 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:04:49.197 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 41, 2021-05-12T12:04:49.197 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:04:49.197 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:02:18.428606+0000", 2021-05-12T12:04:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:03:38.377583+0000", 2021-05-12T12:04:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:03:36.967722+0000", 2021-05-12T12:04:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:03:37.937081+0000", 2021-05-12T12:04:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:04:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:04:49.199 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:04:49.199 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:04:49.199 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:04:49.199 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:04:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:04:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:04:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:04:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:04:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.201 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 9, 2021-05-12T12:04:49.201 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:04:49.201 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:04:49.201 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:04:49.201 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:04:49.202 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:04:49.202 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:04:49.202 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:04:49.202 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:04:49.203 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:04:49.203 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:04:49.203 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.203 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:04:49.204 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:04:49.204 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:02:48.804476+0000", 2021-05-12T12:04:49.204 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 1, 2021-05-12T12:04:49.204 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool", 2021-05-12T12:04:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:04:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:04:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:04:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 0, 2021-05-12T12:04:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:04:49.206 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:04:49.206 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:04:49.206 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:04:49.206 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:04:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:04:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 1, 2021-05-12T12:04:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:04:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 1, 2021-05-12T12:04:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 1, 2021-05-12T12:04:49.208 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 1, 2021-05-12T12:04:49.208 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:04:49.208 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:04:49.208 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:04:49.208 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:04:49.209 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:04:49.209 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:04:49.209 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:04:49.209 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.210 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "28", 2021-05-12T12:04:49.210 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:04:49.210 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:04:49.210 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:04:49.210 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:04:49.211 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:04:49.211 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 0, 2021-05-12T12:04:49.211 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 0, 2021-05-12T12:04:49.211 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [], 2021-05-12T12:04:49.212 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:04:49.212 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:04:49.212 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:04:49.212 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:04:49.212 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:04:49.213 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:04:49.213 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:04:49.213 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:04:49.213 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:04:49.213 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:04:49.214 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:04:49.214 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:04:49.214 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:04:49.214 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:04:49.215 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:04:49.215 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "", 2021-05-12T12:04:49.215 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:04:49.215 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:04:49.215 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:04:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:04:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:04:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:04:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:04:49.217 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:04:49.217 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:04:49.217 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:04:49.217 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 0, 2021-05-12T12:04:49.217 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:04:49.218 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:04:49.218 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:04:49.218 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:04:49.218 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:04:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "f2cb872d-ef34-4fe5-a813-b277592bb9a2", 2021-05-12T12:04:49.220 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:04:49.220 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:49.220 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:49.220 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:04:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:04:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 35, 2021-05-12T12:04:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 39, 2021-05-12T12:04:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 29, 2021-05-12T12:04:49.222 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:49.222 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:49.222 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.222 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.223 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.223 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:04:49.223 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.223 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.223 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.224 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.224 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:04:49.224 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.224 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.224 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.225 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.225 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:49.225 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.225 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.226 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.226 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:04:49.226 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.226 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.226 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.227 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.227 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:04:49.227 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.227 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.227 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.228 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.228 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:49.228 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.228 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.229 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.229 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:04:49.229 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.229 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.229 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.230 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.230 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:04:49.230 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.230 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.230 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.231 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.231 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:49.231 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.231 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.232 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.232 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:04:49.232 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.232 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.232 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.233 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.233 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:04:49.233 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49363 2021-05-12T12:04:49.233 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.233 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.234 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.234 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/49363", 2021-05-12T12:04:49.234 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/49363", 2021-05-12T12:04:49.234 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/49363", 2021-05-12T12:04:49.234 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/49363", 2021-05-12T12:04:49.235 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:49.235 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:04:49.235 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:04:49.235 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.235 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.236 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.236 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:49.236 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "9f1bd5b2-c83e-463c-b023-d48ebd49764a", 2021-05-12T12:04:49.236 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:04:49.237 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:49.237 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:49.237 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:49.237 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:04:49.237 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:04:49.238 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 10, 2021-05-12T12:04:49.238 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 22, 2021-05-12T12:04:49.238 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 30, 2021-05-12T12:04:49.238 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:49.239 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:49.239 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.239 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.239 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.239 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:49.240 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.240 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.240 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.240 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.241 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:49.241 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.241 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.241 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.241 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.242 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:49.242 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.242 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.242 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.242 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:49.243 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.243 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.243 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.243 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.243 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:49.244 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.244 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.244 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.244 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.245 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:49.245 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.245 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.245 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.245 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:49.246 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.246 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.246 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.246 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.246 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:49.247 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.247 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.247 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.247 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.247 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:49.248 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.248 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.248 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.248 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:49.249 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.249 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.249 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.249 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.249 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:49.250 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 43911 2021-05-12T12:04:49.250 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.250 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.250 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.250 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/43911", 2021-05-12T12:04:49.251 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/43911", 2021-05-12T12:04:49.251 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43911", 2021-05-12T12:04:49.251 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43911", 2021-05-12T12:04:49.251 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:49.252 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:04:49.252 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.252 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.252 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.252 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:49.253 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0", 2021-05-12T12:04:49.253 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:04:49.253 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:04:49.253 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:04:49.253 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:04:49.254 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 15, 2021-05-12T12:04:49.254 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:04:49.254 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 37, 2021-05-12T12:04:49.254 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:04:49.255 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 31, 2021-05-12T12:04:49.255 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:04:49.255 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:04:49.255 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.255 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.256 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.256 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:04:49.256 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.256 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.257 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.257 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.257 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:04:49.257 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.257 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.258 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.258 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.258 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:04:49.258 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.259 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.259 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.259 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:04:49.259 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.259 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.260 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.260 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.260 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:04:49.260 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.260 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.261 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.261 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.261 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:04:49.261 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.262 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.262 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.262 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:04:49.262 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.262 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.263 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.263 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.263 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:04:49.263 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.264 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.264 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.264 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.264 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:04:49.264 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:04:49.265 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.265 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:04:49.265 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:04:49.265 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.265 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.266 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.266 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:04:49.266 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:04:49.266 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 49920 2021-05-12T12:04:49.267 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.267 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.267 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.267 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/49920", 2021-05-12T12:04:49.268 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/49920", 2021-05-12T12:04:49.268 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/49920", 2021-05-12T12:04:49.268 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/49920", 2021-05-12T12:04:49.268 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:04:49.268 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:04:49.269 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:04:49.269 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.269 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.269 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.269 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:04:49.270 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.270 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:49.270 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:27.325634+0000", 2021-05-12T12:04:49.270 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:49.271 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:49.271 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:49.271 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:49.271 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:28.169537+0000", 2021-05-12T12:04:49.271 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:49.272 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.272 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.272 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:49.272 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:27.838193+0000", 2021-05-12T12:04:49.273 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:49.273 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:49.273 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:49.273 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:49.273 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:37.353029+0000", 2021-05-12T12:04:49.274 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:49.274 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.274 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.274 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:49.275 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:03:28.348696+0000", 2021-05-12T12:04:49.275 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:04:49.275 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:04:49.275 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:04:49.276 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:04:49.276 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:02:46.582993+0000", 2021-05-12T12:04:49.276 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:04:49.276 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.276 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.277 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:04:49.277 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:04:49.277 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:04:49.277 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:04:49.277 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:04:49.278 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:04:49.278 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:04:49.278 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:04:49.278 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:04:49.278 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:04:49.279 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:04:49.279 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.279 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.279 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:04:49.280 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:04:49.280 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:04:49.280 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:04:49.280 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:04:49.280 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:04:49.281 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:04:49.281 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:04:49.281 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:04:49.281 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:04:49.281 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:04:49.282 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.282 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.282 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:04:49.282 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.283 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:49.283 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:49.283 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/49363,v1:127.0.0.1:6805/49363]", 2021-05-12T12:04:49.283 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:49.284 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:49.284 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:49.284 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:49.284 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:49.284 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:49.285 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:49.285 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:49.285 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:49.285 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:04:49.285 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:49.286 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:49.286 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:49.286 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:49.286 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:49.287 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:49.287 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:49.287 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:49.287 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:49.287 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:49.288 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:49.288 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:49.288 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:49.288 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:49.288 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:49.289 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/49363,v1:127.0.0.1:6803/49363]", 2021-05-12T12:04:49.289 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:49.289 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/49363,v1:127.0.0.1:6809/49363]", 2021-05-12T12:04:49.289 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/49363,v1:127.0.0.1:6807/49363]", 2021-05-12T12:04:49.290 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:49.290 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:49.290 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:49.290 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:49.290 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:49.291 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:49.291 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:49.291 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:49.291 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:49.292 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:04:49.292 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:49.292 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:49.292 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:49.292 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.293 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.293 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:49.293 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:49.293 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/43911,v1:127.0.0.1:6813/43911]", 2021-05-12T12:04:49.293 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:49.294 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:49.294 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:49.294 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:49.294 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:49.295 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:49.295 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:49.295 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:49.295 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:49.295 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:04:49.296 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:49.296 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:49.296 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:49.296 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:49.297 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:49.297 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:49.297 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:49.297 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:49.297 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:49.298 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:49.298 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:49.298 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:49.298 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:49.299 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:49.299 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:49.299 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/43911,v1:127.0.0.1:6811/43911]", 2021-05-12T12:04:49.299 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:49.299 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43911,v1:127.0.0.1:6817/43911]", 2021-05-12T12:04:49.300 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43911,v1:127.0.0.1:6815/43911]", 2021-05-12T12:04:49.300 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:49.300 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:49.300 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:49.301 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:49.301 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:49.301 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:49.301 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:49.301 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:04:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:04:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/49920,v1:127.0.0.1:6813/49920]", 2021-05-12T12:04:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:04:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:04:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:04:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:04:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:04:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:04:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:04:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:04:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:04:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:04:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:04:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:04:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:04:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:04:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:04:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:04:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:04:49.308 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:04:49.308 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:04:49.308 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:04:49.308 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:04:49.309 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:04:49.309 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:04:49.309 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:04:49.309 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:04:49.309 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/49920,v1:127.0.0.1:6811/49920]", 2021-05-12T12:04:49.310 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:04:49.310 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/49920,v1:127.0.0.1:6817/49920]", 2021-05-12T12:04:49.310 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/49920,v1:127.0.0.1:6815/49920]", 2021-05-12T12:04:49.310 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:04:49.310 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:04:49.311 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:04:49.311 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:04:49.311 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:04:49.311 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:04:49.312 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:04:49.312 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:04:49.312 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:04:49.312 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:04:49.312 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:04:49.313 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:04:49.313 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:04:49.313 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.313 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.314 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:04:49.314 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:04:49.314 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:04:49.314 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:04:49.314 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.315 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:04:49.315 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.315 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:49.315 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 35 2021-05-12T12:04:49.316 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.316 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.316 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:49.316 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 10 2021-05-12T12:04:49.316 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.317 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.317 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:49.317 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 38 2021-05-12T12:04:49.317 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.318 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.318 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.318 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:04:49.318 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 41, 2021-05-12T12:04:49.318 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:04:49.319 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:04:49.319 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.319 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:49.319 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:04:49.319 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:49.320 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.320 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.320 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:49.320 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:04:49.321 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:49.321 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.321 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.321 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:49.321 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:04:49.322 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:04:49.322 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.322 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.322 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:04:49.323 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.323 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:04:49.323 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:04:49.323 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.323 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.324 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:49.324 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:04:49.324 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.324 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.325 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:04:49.325 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:04:49.325 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.325 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.325 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:04:49.326 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:04:49.326 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.326 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.326 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:04:49.326 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:04:49.327 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.327 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.327 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:04:49.327 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:04:49.328 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.328 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.328 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:04:49.328 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:04:49.328 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.329 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.329 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:04:49.329 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:04:49.329 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.329 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.330 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:04:49.330 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:04:49.330 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.330 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.331 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:04:49.331 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:04:49.331 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.331 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.331 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:04:49.332 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:04:49.332 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.332 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.332 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:49.333 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:04:49.333 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.333 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.333 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:04:49.333 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.334 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:04:49.334 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:04:49.334 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:49.334 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:04:49.335 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.335 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:49.335 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:49.335 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:49.335 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.336 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:04:49.336 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.336 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:49.336 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.336 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.337 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.337 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.337 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:04:49.337 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:04:49.337 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:04:49.338 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:04:49.338 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.338 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:49.338 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:49.339 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:49.339 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.339 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:04:49.339 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.339 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:49.340 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.340 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.340 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.340 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.340 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:04:49.341 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:04:49.341 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:49.341 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:04:49.341 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.341 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:49.342 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:49.342 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:49.342 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.342 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:49.343 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.343 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:49.343 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.343 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.343 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:49.344 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.344 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:04:49.344 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.344 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.344 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:49.345 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.345 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:04:49.345 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.345 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.345 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.346 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.346 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:04:49.346 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:04:49.346 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:04:49.346 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:04:49.347 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:04:49.347 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:04:49.347 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:04:49.347 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:04:49.348 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.348 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:04:49.348 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.348 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:04:49.348 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.349 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.349 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:04:49.349 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.349 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:04:49.349 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.350 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.350 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:04:49.350 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:04:49.350 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:04:49.350 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.351 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.351 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.351 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.351 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:04:49.351 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.352 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:04:49.352 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:04:49.352 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:04:49.352 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:04:49.353 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:04:49.353 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:04:49.353 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:04:49.353 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.353 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:04:49.354 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:04:49.354 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:04:49.354 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.354 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.354 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:04:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:04:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:04:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:04:49.356 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.356 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:04:49.356 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.356 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:04:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:04:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:04:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:04:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:04:49.358 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:04:49.358 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:04:49.358 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:04:49.358 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:04:49.358 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:04:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:04:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:04:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:04:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:04:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:04:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 0, 2021-05-12T12:04:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:04:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:04:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:04:49.361 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:04:49.361 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:04:49.361 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.361 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:04:49.361 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.362 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:04:49.362 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:04:49.362 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:04:49.362 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:04:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:04:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:04:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:04:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:04:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:04:49.364 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:04:49.364 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:04:49.364 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:04:49.364 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:04:49.364 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:04:49.365 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:04:49.365 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:04:49.365 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.365 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.366 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:04:49.366 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:04:49.366 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:04:49.366 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.366 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:04:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:04:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:04:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:04:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:04:49.368 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:04:49.368 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:04:49.368 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:04:49.368 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.369 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:04:49.369 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 1, 2021-05-12T12:04:49.369 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:04:49.369 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:04:49.369 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:04:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:04:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 47947, 2021-05-12T12:04:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:04:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:04:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:04:49.371 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:49.371 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:04:49.371 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:49.371 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:49.371 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:49.372 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:49.372 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:04:49.372 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:04:49.372 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:04:49.373 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:04:49.373 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:04:49.373 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:49.373 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:49.373 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:49.374 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:49.374 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:49.374 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:49.374 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:04:49.374 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:04:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:04:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:04:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:04:49.376 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:04:49.376 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:49.376 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:49.376 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:49.378 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:49.378 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:04:49.378 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:04:49.378 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.378 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:04:49.379 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:04:49.379 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:04:49.379 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:49.379 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:04:49.380 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:04:49.380 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:49.380 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:49.380 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:49.380 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:04:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:04:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 17, 2021-05-12T12:04:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 17, 2021-05-12T12:04:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:04:49.382 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:04:49.382 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:04:49.382 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.382 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:04:49.382 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:04:49.383 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:04:49.383 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:04:49.383 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:04:49.383 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:49.383 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:49.384 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:04:49.384 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18448, 2021-05-12T12:04:49.384 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2096, 2021-05-12T12:04:49.384 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:04:49.384 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:49.385 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554352, 2021-05-12T12:04:49.385 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:04:49.385 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:04:49.385 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103656448, 2021-05-12T12:04:49.386 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:49.386 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2146304, 2021-05-12T12:04:49.386 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 414314, 2021-05-12T12:04:49.386 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:49.386 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:49.387 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:49.387 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:04:49.387 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:04:49.387 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.387 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:04:49.388 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:49.388 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:49.388 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:49.388 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:04:49.388 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:04:49.389 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:04:49.389 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.389 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:04:49.389 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:49.389 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:49.390 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:49.390 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:04:49.390 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.390 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:04:49.390 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:04:49.391 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.391 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:04:49.391 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:04:49.391 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:04:49.391 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:04:49.392 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:04:49.392 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:04:49.392 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:04:49.392 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:04:49.393 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:04:49.393 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18448, 2021-05-12T12:04:49.393 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2096, 2021-05-12T12:04:49.393 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:04:49.393 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16317, 2021-05-12T12:04:49.394 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554352, 2021-05-12T12:04:49.394 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:04:49.394 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:04:49.394 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103656448, 2021-05-12T12:04:49.394 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:49.395 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2146304, 2021-05-12T12:04:49.395 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 414314, 2021-05-12T12:04:49.395 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:49.395 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:49.396 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:49.396 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:04:49.396 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16709520 2021-05-12T12:04:49.396 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.396 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:04:49.397 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:04:49.397 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:04:49.397 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:04:49.397 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:04:49.397 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:04:49.398 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:04:49.398 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.398 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:04:49.398 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:04:49.398 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:04:49.399 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:04:49.399 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:04:49.399 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.399 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:04:49.399 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:04:49.400 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.400 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.400 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:04:49.400 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.401 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:04:49.401 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:04:49.401 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:04:49.401 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 47947, 2021-05-12T12:04:49.401 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:04:49.402 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:04:49.402 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:04:49.402 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:04:49.402 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:04:49.402 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:04:49.403 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:04:49.403 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:04:49.403 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:04:49.403 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:04:49.403 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:04:49.404 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:04:49.404 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:04:49.404 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:04:49.404 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:04:49.405 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:04:49.405 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:04:49.405 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:04:49.405 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:04:49.405 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:04:49.406 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:04:49.406 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:04:49.406 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:04:49.406 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:04:49.406 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:04:49.407 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:04:49.407 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:04:49.407 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:04:49.407 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:04:49.408 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:04:49.408 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:04:49.408 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:04:49.408 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:04:49.408 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:04:49.409 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:04:49.409 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:04:49.409 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:04:49.409 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:04:49.409 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:04:49.410 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.410 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:04:49.410 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:04:49.410 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:04:49.411 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:04:49.411 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1220608, 2021-05-12T12:04:49.411 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 142522, 2021-05-12T12:04:49.411 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:04:49.411 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:04:49.412 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:04:49.412 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:04:49.412 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:04:49.412 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.412 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 17, 2021-05-12T12:04:49.413 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 17, 2021-05-12T12:04:49.413 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 2, 2021-05-12T12:04:49.413 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 2, 2021-05-12T12:04:49.413 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:04:49.414 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.414 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.414 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:04:49.414 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.415 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:49.415 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 150323855379 2021-05-12T12:04:49.415 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.415 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.415 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:49.416 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 42949672976 2021-05-12T12:04:49.416 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.416 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.416 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:49.416 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 158913789971 2021-05-12T12:04:49.417 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.417 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.417 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:04:49.417 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.418 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:04:49.418 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 1 2021-05-12T12:04:49.418 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.418 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.418 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:04:49.419 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.419 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:04:49.419 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:04:49.419 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:04:49.420 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:49.420 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.420 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.420 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:04:49.420 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:04:49.421 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:04:49.421 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:49.421 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.421 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.421 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:04:49.422 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 1, 2021-05-12T12:04:49.422 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:04:49.422 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:04:49.422 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.423 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.423 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:04:49.423 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:04:49.423 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:04:49.423 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:04:49.424 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.424 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:04:49.424 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:04:49.424 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 6, 2021-05-12T12:04:49.425 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:04:37.394588+0000", 2021-05-12T12:04:49.425 INFO:tasks.workunit.client.0.smithi090.stdout: "services": {} 2021-05-12T12:04:49.425 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:04:49.425 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:04:49.425 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:04:49.426 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 452, 2021-05-12T12:04:49.426 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:04:49.426 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:04:49.426 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:04:49.426 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:04:49.427 INFO:tasks.workunit.client.0.smithi090.stderr:report 3997860363 2021-05-12T12:04:49.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:04:49.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2021-05-12T12:04:49.428 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery 2021-05-12T12:04:49.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:49.521 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:49.522 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 41'101 41:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:49.522 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:49.522 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:49.523 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:49.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:04:49.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2021-05-12T12:04:49.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:04:49.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:04:49.534 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:49.535 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:49.535 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:04:49.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:04:49.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:49.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:04:49.628 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:49.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T12:04:49.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2021-05-12T12:04:49.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:04:49.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:04:49.638 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:49.638 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:49.638 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:04:49.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:04:49.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:49.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:04:49.731 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:49.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2021-05-12T12:04:49.739 INFO:tasks.workunit.client.0.smithi090.stdout:revive divergent 1 2021-05-12T12:04:49.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:50.080 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:50.080 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 41'101 41:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:50.081 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:50.081 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:50.082 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:50.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2021-05-12T12:04:50.679 INFO:tasks.workunit.client.0.smithi090.stderr:noup is set 2021-05-12T12:04:50.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-05-12T12:04:50.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:04:50.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:04:50.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:04:50.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:04:50.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:04:50.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:04:50.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:04:50.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:04:50.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:04:50.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:04:50.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:04:50.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:04:50.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:04:50.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:04:50.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:04:50.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:04:50.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:04:50.695 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:04:50.695 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:50.695 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:04:50.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:04:50.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:04:50.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:04:50.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:04:50.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:04:50.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:04:50.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:04:50.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:50.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:04:50.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:04:50.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:04:50.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:04:50.698 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:04:50.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:04:50.699 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:04:50.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:04:50.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:04:50.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:04:50.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:04:50.718 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:04:50.716+0000 7fbe3071a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:50.726 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:04:50.724+0000 7fbe3071a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:50.727 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:04:50.725+0000 7fbe3071a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:04:51.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:04:51.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2021-05-12T12:04:51.243 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:04:51.241+0000 7fbe3071a0c0 -1 Falling back to public interface 2021-05-12T12:04:53.902 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:04:53.900+0000 7fbe3071a0c0 -1 osd.1 28 log_to_monitors {default=true} 2021-05-12T12:04:56.063 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery divergent 2021-05-12T12:04:56.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2021-05-12T12:04:56.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:56.413 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:56.414 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 41'101 42:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:56.414 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:56.414 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:56.415 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:56.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2021-05-12T12:04:56.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:04:56.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:04:56.426 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:04:56.426 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:04:56.426 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:04:56.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:04:56.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:04:56.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:04:56.520 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:04:56.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2021-05-12T12:04:57.118 INFO:tasks.workunit.client.0.smithi090.stderr:noup is unset 2021-05-12T12:04:57.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2021-05-12T12:04:57.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:57.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:04:57.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:57.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:57.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:57.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:57.129 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:04:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:04:57.595 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/49363,v1:127.0.0.1:6803/49363] [v2:127.0.0.1:6804/49363,v1:127.0.0.1:6805/49363] exists,up f2cb872d-ef34-4fe5-a813-b277592bb9a2 2021-05-12T12:04:57.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:57.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:57.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:57.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2021-05-12T12:04:57.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:57.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:04:57.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:57.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:57.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:57.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:57.599 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:04:57.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:57.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:04:57.958 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/52610,v1:127.0.0.1:6819/52610] [v2:127.0.0.1:6820/52610,v1:127.0.0.1:6821/52610] exists,up 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:04:57.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:57.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:57.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:57.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2021-05-12T12:04:57.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:04:57.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:04:57.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:04:57.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:04:57.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:04:57.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:04:57.962 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:04:57.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:04:57.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:04:58.322 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/49920,v1:127.0.0.1:6811/49920] [v2:127.0.0.1:6812/49920,v1:127.0.0.1:6813/49920] exists,up c396f1ac-1990-49fb-b2ba-bd4b9e84d9b0 2021-05-12T12:04:58.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:04:58.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:04:58.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:04:58.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:58.662 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:58.662 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 41'101 42:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:58.662 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:58.663 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:58.663 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:58.672 INFO:tasks.workunit.client.0.smithi090.stdout:wait for peering 2021-05-12T12:04:58.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2021-05-12T12:04:58.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:04:59.011 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:04:59.011 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T12:03:38.710439+0000 41'101 42:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:04:59.012 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:04:59.012 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:04:59.013 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:04:59.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2021-05-12T12:05:06.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2021-05-12T12:05:06.819 INFO:tasks.workunit.client.0.smithi090.stdout:killing divergent 1 2021-05-12T12:05:06.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:07.167 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:07.168 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T12:04:57.759920+0000 41'101 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:05:07.168 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:05:07.168 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:07.169 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:05:07.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:05:07.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:07.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:07.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:07.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:07.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:07.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:07.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.29659.out 2021-05-12T12:05:07.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.29659.out 2021-05-12T12:05:07.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:05:07.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:05:07.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:05:07.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:05:07.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:05:07.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.29659.out 2021-05-12T12:05:08.120 INFO:tasks.workunit.client.0.smithi090.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=22/22 lis/c=46/22 les/c/f=47/23/0 sis=46 pruub=8.953603745s) 2021-05-12T12:05:08.125 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:023dd3d2:::existing_12:head# 2021-05-12T12:05:08.141 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:04c40a0c:::existing_93:head# 2021-05-12T12:05:08.159 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:064ff5fb:::existing_36:head# 2021-05-12T12:05:08.162 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:091f9f1d:::existing_94:head# 2021-05-12T12:05:08.168 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:0a865a93:::existing_54:head# 2021-05-12T12:05:08.168 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:0fed513c:::existing_41:head# 2021-05-12T12:05:08.169 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:13b9e0b8:::existing_92:head# 2021-05-12T12:05:08.169 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:169157fa:::existing_81:head# 2021-05-12T12:05:08.171 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:17731382:::existing_58:head# 2021-05-12T12:05:08.171 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:18a45a39:::existing_43:head# 2021-05-12T12:05:08.171 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:1c160f01:::existing_66:head# 2021-05-12T12:05:08.171 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:1dd6c249:::existing_97:head# 2021-05-12T12:05:08.172 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:227e0c77:::existing_7:head# 2021-05-12T12:05:08.176 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:23e1b063:::existing_59:head# 2021-05-12T12:05:08.177 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:28566edb:::existing_78:head# 2021-05-12T12:05:08.177 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:2a101182:::existing_20:head# 2021-05-12T12:05:08.177 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:2d1519da:::existing_96:head# 2021-05-12T12:05:08.177 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:3078b87a:::existing_86:head# 2021-05-12T12:05:08.178 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:30bde01b:::existing_15:head# 2021-05-12T12:05:08.178 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:317a7e19:::existing_56:head# 2021-05-12T12:05:08.178 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:38b9c7fa:::existing_33:head# 2021-05-12T12:05:08.178 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:3e16d84b:::existing_48:head# 2021-05-12T12:05:08.178 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:45fe8203:::existing_75:head# 2021-05-12T12:05:08.179 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:4e82c9ba:::existing_50:head# 2021-05-12T12:05:08.179 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:5298dcdb:::existing_57:head# 2021-05-12T12:05:08.179 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:52ebaa44:::existing_32:head# 2021-05-12T12:05:08.179 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:53588912:::existing_9:head# 2021-05-12T12:05:08.180 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:5b7e2de6:::existing_84:head# 2021-05-12T12:05:08.180 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:5da0255c:::existing_13:head# 2021-05-12T12:05:08.180 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:5da2dab8:::existing_53:head# 2021-05-12T12:05:08.180 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:5ee3514b:::existing_30:head# 2021-05-12T12:05:08.180 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:602f83fe:::foo:head# 2021-05-12T12:05:08.181 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:60426cc7:::existing_62:head# 2021-05-12T12:05:08.181 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:628918ca:::existing_79:head# 2021-05-12T12:05:08.181 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:63f8ab2d:::existing_77:head# 2021-05-12T12:05:08.181 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:67e1780f:::existing_27:head# 2021-05-12T12:05:08.181 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:68143ca1:::existing_71:head# 2021-05-12T12:05:08.193 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:6bac9509:::existing_29:head# 2021-05-12T12:05:08.193 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:6bbec798:::existing_45:head# 2021-05-12T12:05:08.194 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:6cb23a40:::existing_65:head# 2021-05-12T12:05:08.194 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:7055ec5e:::existing_83:head# 2021-05-12T12:05:08.194 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:70a54e2b:::existing_87:head# 2021-05-12T12:05:08.194 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:750ace21:::existing_19:head# 2021-05-12T12:05:08.195 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:78623b1d:::existing_6:head# 2021-05-12T12:05:08.195 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:79881152:::existing_63:head# 2021-05-12T12:05:08.195 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:7a80c0d5:::existing_99:head# 2021-05-12T12:05:08.195 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:81490677:::existing_76:head# 2021-05-12T12:05:08.195 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:85ec8e5d:::existing_85:head# 2021-05-12T12:05:08.196 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:8e2abbf0:::existing_49:head# 2021-05-12T12:05:08.196 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:8ee0fe0f:::existing_52:head# 2021-05-12T12:05:08.196 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:923c5d4d:::existing_5:head# 2021-05-12T12:05:08.196 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:92731e71:::existing_89:head# 2021-05-12T12:05:08.197 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:928205cf:::existing_82:head# 2021-05-12T12:05:08.197 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:93be2267:::existing_24:head# 2021-05-12T12:05:08.197 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:99ceda2c:::existing_44:head# 2021-05-12T12:05:08.197 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:9dd917ee:::existing_26:head# 2021-05-12T12:05:08.197 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a28805a6:::existing_72:head# 2021-05-12T12:05:08.198 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a294a8f6:::existing_16:head# 2021-05-12T12:05:08.198 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a2f5473a:::existing_42:head# 2021-05-12T12:05:08.198 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a504a533:::existing_90:head# 2021-05-12T12:05:08.198 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a691337e:::existing_61:head# 2021-05-12T12:05:08.199 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a6e8d7ac:::existing_28:head# 2021-05-12T12:05:08.199 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:a9fde86e:::existing_18:head# 2021-05-12T12:05:08.199 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:aa309faa:::existing_68:head# 2021-05-12T12:05:08.199 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:ae40b0ed:::existing_31:head# 2021-05-12T12:05:08.199 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:b279b1fe:::existing_10:head# 2021-05-12T12:05:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:b343e277:::existing_88:head# 2021-05-12T12:05:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:b7130e90:::existing_80:head# 2021-05-12T12:05:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:bc6f7309:::existing_70:head# 2021-05-12T12:05:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:be8489b7:::existing_38:head# 2021-05-12T12:05:08.200 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:beb4e5e3:::existing_47:head# 2021-05-12T12:05:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:bebec877:::existing_74:head# 2021-05-12T12:05:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:c04347d8:::existing_60:head# 2021-05-12T12:05:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:c156033e:::existing_100:head# 2021-05-12T12:05:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:c19f5a8e:::existing_8:head# 2021-05-12T12:05:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:c3e6ebe4:::existing_17:head# 2021-05-12T12:05:08.202 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:c6c5a7c2:::existing_39:head# 2021-05-12T12:05:08.202 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:cc1d0abf:::existing_95:head# 2021-05-12T12:05:08.202 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:ccb3865b:::existing_73:head# 2021-05-12T12:05:08.202 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:cd84e19e:::existing_40:head# 2021-05-12T12:05:08.203 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:d2cd64ce:::existing_37:head# 2021-05-12T12:05:08.203 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:d41a25c9:::existing_22:head# 2021-05-12T12:05:08.203 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:d7f088da:::existing_91:head# 2021-05-12T12:05:08.203 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:d9cbd1d6:::existing_64:head# 2021-05-12T12:05:08.204 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:dcc4ff82:::existing_98:head# 2021-05-12T12:05:08.204 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:dd677bd3:::existing_35:head# 2021-05-12T12:05:08.204 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:dd6d90e7:::existing_46:head# 2021-05-12T12:05:08.204 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:e0e22077:::existing_21:head# 2021-05-12T12:05:08.204 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:e255b710:::existing_55:head# 2021-05-12T12:05:08.205 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:eff1d5db:::existing_14:head# 2021-05-12T12:05:08.205 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:f1b156bf:::existing_23:head# 2021-05-12T12:05:08.205 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:f5ce4199:::existing_34:head# 2021-05-12T12:05:08.205 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:f6809456:::existing_11:head# 2021-05-12T12:05:08.206 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:f699a2a9:::existing_67:head# 2021-05-12T12:05:08.206 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:f9baac43:::existing_25:head# 2021-05-12T12:05:08.206 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:fb76688e:::existing_51:head# 2021-05-12T12:05:08.206 INFO:tasks.workunit.client.0.smithi090.stderr:Read #1:fbb92eec:::existing_69:head# 2021-05-12T12:05:08.207 INFO:tasks.workunit.client.0.smithi090.stderr:Export successful 2021-05-12T12:05:08.207 INFO:tasks.workunit.client.0.smithi090.stdout: marking collection for removal 2021-05-12T12:05:08.207 INFO:tasks.workunit.client.0.smithi090.stdout:setting '_remove' omap key 2021-05-12T12:05:08.208 INFO:tasks.workunit.client.0.smithi090.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:05:08.437 INFO:tasks.workunit.client.0.smithi090.stderr:Remove successful 2021-05-12T12:05:08.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.29659.out 2021-05-12T12:05:08.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:05:08.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:05:08.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:05:08.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:05:08.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:05:08.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.29659.out 2021-05-12T12:05:09.525 INFO:tasks.workunit.client.0.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={22=1}} deleted_pools ) 2021-05-12T12:05:09.525 INFO:tasks.workunit.client.0.smithi090.stdout:Importing pgid 1.0 2021-05-12T12:05:09.526 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:023dd3d2:::existing_12:head# 2021-05-12T12:05:09.526 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.600 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:04c40a0c:::existing_93:head# 2021-05-12T12:05:09.600 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.625 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:064ff5fb:::existing_36:head# 2021-05-12T12:05:09.625 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.650 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:091f9f1d:::existing_94:head# 2021-05-12T12:05:09.650 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.675 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:0a865a93:::existing_54:head# 2021-05-12T12:05:09.676 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.700 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:0fed513c:::existing_41:head# 2021-05-12T12:05:09.700 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.726 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:13b9e0b8:::existing_92:head# 2021-05-12T12:05:09.726 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.784 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:169157fa:::existing_81:head# 2021-05-12T12:05:09.785 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.809 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:17731382:::existing_58:head# 2021-05-12T12:05:09.809 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.834 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:18a45a39:::existing_43:head# 2021-05-12T12:05:09.835 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.859 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:1c160f01:::existing_66:head# 2021-05-12T12:05:09.860 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.884 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:1dd6c249:::existing_97:head# 2021-05-12T12:05:09.885 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.909 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:227e0c77:::existing_7:head# 2021-05-12T12:05:09.910 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.943 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:23e1b063:::existing_59:head# 2021-05-12T12:05:09.944 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:09.968 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:28566edb:::existing_78:head# 2021-05-12T12:05:09.969 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.035 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:2a101182:::existing_20:head# 2021-05-12T12:05:10.036 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.086 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:2d1519da:::existing_96:head# 2021-05-12T12:05:10.086 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.111 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:3078b87a:::existing_86:head# 2021-05-12T12:05:10.111 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.136 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:30bde01b:::existing_15:head# 2021-05-12T12:05:10.136 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.161 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:317a7e19:::existing_56:head# 2021-05-12T12:05:10.162 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.186 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:38b9c7fa:::existing_33:head# 2021-05-12T12:05:10.187 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.211 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:3e16d84b:::existing_48:head# 2021-05-12T12:05:10.212 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.237 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:45fe8203:::existing_75:head# 2021-05-12T12:05:10.237 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.262 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:4e82c9ba:::existing_50:head# 2021-05-12T12:05:10.262 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.287 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:5298dcdb:::existing_57:head# 2021-05-12T12:05:10.287 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.312 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:52ebaa44:::existing_32:head# 2021-05-12T12:05:10.312 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.337 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:53588912:::existing_9:head# 2021-05-12T12:05:10.338 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.362 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:5b7e2de6:::existing_84:head# 2021-05-12T12:05:10.363 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.387 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:5da0255c:::existing_13:head# 2021-05-12T12:05:10.388 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.413 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:5da2dab8:::existing_53:head# 2021-05-12T12:05:10.413 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.438 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:5ee3514b:::existing_30:head# 2021-05-12T12:05:10.438 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.472 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:602f83fe:::foo:head# 2021-05-12T12:05:10.472 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.505 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:60426cc7:::existing_62:head# 2021-05-12T12:05:10.505 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.538 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:628918ca:::existing_79:head# 2021-05-12T12:05:10.539 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.572 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:63f8ab2d:::existing_77:head# 2021-05-12T12:05:10.572 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.605 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:67e1780f:::existing_27:head# 2021-05-12T12:05:10.605 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.639 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:68143ca1:::existing_71:head# 2021-05-12T12:05:10.639 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.672 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:6bac9509:::existing_29:head# 2021-05-12T12:05:10.672 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.739 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:6bbec798:::existing_45:head# 2021-05-12T12:05:10.739 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.772 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:6cb23a40:::existing_65:head# 2021-05-12T12:05:10.773 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.806 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:7055ec5e:::existing_83:head# 2021-05-12T12:05:10.806 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.839 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:70a54e2b:::existing_87:head# 2021-05-12T12:05:10.840 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.873 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:750ace21:::existing_19:head# 2021-05-12T12:05:10.873 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.906 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:78623b1d:::existing_6:head# 2021-05-12T12:05:10.907 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.940 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:79881152:::existing_63:head# 2021-05-12T12:05:10.940 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:10.975 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:7a80c0d5:::existing_99:head# 2021-05-12T12:05:10.975 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.000 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:81490677:::existing_76:head# 2021-05-12T12:05:11.000 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.025 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:85ec8e5d:::existing_85:head# 2021-05-12T12:05:11.025 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.050 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:8e2abbf0:::existing_49:head# 2021-05-12T12:05:11.050 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.075 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:8ee0fe0f:::existing_52:head# 2021-05-12T12:05:11.076 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.100 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:923c5d4d:::existing_5:head# 2021-05-12T12:05:11.101 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.126 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:92731e71:::existing_89:head# 2021-05-12T12:05:11.126 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.151 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:928205cf:::existing_82:head# 2021-05-12T12:05:11.151 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.176 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:93be2267:::existing_24:head# 2021-05-12T12:05:11.176 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.201 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:99ceda2c:::existing_44:head# 2021-05-12T12:05:11.202 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.226 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:9dd917ee:::existing_26:head# 2021-05-12T12:05:11.227 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.259 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a28805a6:::existing_72:head# 2021-05-12T12:05:11.260 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.293 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a294a8f6:::existing_16:head# 2021-05-12T12:05:11.293 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.327 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a2f5473a:::existing_42:head# 2021-05-12T12:05:11.327 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.377 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a504a533:::existing_90:head# 2021-05-12T12:05:11.377 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.427 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a691337e:::existing_61:head# 2021-05-12T12:05:11.428 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.487 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a6e8d7ac:::existing_28:head# 2021-05-12T12:05:11.487 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.580 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:a9fde86e:::existing_18:head# 2021-05-12T12:05:11.580 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.629 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:aa309faa:::existing_68:head# 2021-05-12T12:05:11.630 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.704 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:ae40b0ed:::existing_31:head# 2021-05-12T12:05:11.704 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.763 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:b279b1fe:::existing_10:head# 2021-05-12T12:05:11.763 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.813 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:b343e277:::existing_88:head# 2021-05-12T12:05:11.813 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.863 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:b7130e90:::existing_80:head# 2021-05-12T12:05:11.864 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.914 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:bc6f7309:::existing_70:head# 2021-05-12T12:05:11.914 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:11.964 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:be8489b7:::existing_38:head# 2021-05-12T12:05:11.964 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:beb4e5e3:::existing_47:head# 2021-05-12T12:05:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.114 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:bebec877:::existing_74:head# 2021-05-12T12:05:12.115 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.165 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:c04347d8:::existing_60:head# 2021-05-12T12:05:12.165 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.215 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:c156033e:::existing_100:head# 2021-05-12T12:05:12.215 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.265 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:c19f5a8e:::existing_8:head# 2021-05-12T12:05:12.266 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.316 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:c3e6ebe4:::existing_17:head# 2021-05-12T12:05:12.316 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.366 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:c6c5a7c2:::existing_39:head# 2021-05-12T12:05:12.367 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.409 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:cc1d0abf:::existing_95:head# 2021-05-12T12:05:12.410 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.443 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:ccb3865b:::existing_73:head# 2021-05-12T12:05:12.443 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.485 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:cd84e19e:::existing_40:head# 2021-05-12T12:05:12.485 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.527 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:d2cd64ce:::existing_37:head# 2021-05-12T12:05:12.527 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.569 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:d41a25c9:::existing_22:head# 2021-05-12T12:05:12.569 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:d7f088da:::existing_91:head# 2021-05-12T12:05:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.678 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:d9cbd1d6:::existing_64:head# 2021-05-12T12:05:12.678 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.728 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:dcc4ff82:::existing_98:head# 2021-05-12T12:05:12.728 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.778 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:dd677bd3:::existing_35:head# 2021-05-12T12:05:12.779 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.829 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:dd6d90e7:::existing_46:head# 2021-05-12T12:05:12.829 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.879 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:e0e22077:::existing_21:head# 2021-05-12T12:05:12.879 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.929 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:e255b710:::existing_55:head# 2021-05-12T12:05:12.929 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:12.980 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:eff1d5db:::existing_14:head# 2021-05-12T12:05:12.980 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.030 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:f1b156bf:::existing_23:head# 2021-05-12T12:05:13.030 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.080 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:f5ce4199:::existing_34:head# 2021-05-12T12:05:13.080 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.130 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:f6809456:::existing_11:head# 2021-05-12T12:05:13.131 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.181 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:f699a2a9:::existing_67:head# 2021-05-12T12:05:13.181 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.231 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:f9baac43:::existing_25:head# 2021-05-12T12:05:13.231 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.281 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:fb76688e:::existing_51:head# 2021-05-12T12:05:13.282 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.332 INFO:tasks.workunit.client.0.smithi090.stdout:Write #1:fbb92eec:::existing_69:head# 2021-05-12T12:05:13.332 INFO:tasks.workunit.client.0.smithi090.stdout:snapset 0=[]:{} 2021-05-12T12:05:13.365 INFO:tasks.workunit.client.0.smithi090.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=22/22 lis/c=46/22 les/c/f=47/23/0 sis=46 pruub=8.953603745s) 2021-05-12T12:05:13.415 INFO:tasks.workunit.client.0.smithi090.stdout:Import successful 2021-05-12T12:05:13.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2021-05-12T12:05:13.935 INFO:tasks.workunit.client.0.smithi090.stdout:reviving divergent 1 2021-05-12T12:05:13.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:14.290 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:14.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T12:04:57.759920+0000 41'101 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:05:14.291 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:05:14.291 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:14.292 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:05:14.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-05-12T12:05:14.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:05:14.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:05:14.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:05:14.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:05:14.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:05:14.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:14.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:14.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:14.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:14.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:05:14.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:05:14.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:14.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:05:14.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:14.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:05:14.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:14.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:14.309 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:14.309 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:14.310 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:14.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:14.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:14.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:14.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:14.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:14.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:14.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:14.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:14.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:05:14.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:05:14.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:05:14.313 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:05:14.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:05:14.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:14.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:05:14.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:05:14.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:05:14.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:05:14.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:05:14.332 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:14.329+0000 7f32b8e8b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:14.334 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:14.331+0000 7f32b8e8b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:14.335 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:14.332+0000 7f32b8e8b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:14.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:05:14.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:14.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:05:14.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:14.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:14.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:14.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:14.674 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:05:14.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:14.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:14.852 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:14.849+0000 7f32b8e8b0c0 -1 Falling back to public interface 2021-05-12T12:05:15.039 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/52610,v1:127.0.0.1:6819/52610] [v2:127.0.0.1:6820/52610,v1:127.0.0.1:6821/52610] exists,up 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:05:15.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:15.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:15.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:15.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2021-05-12T12:05:15.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:15.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:05:15.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:15.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:15.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:15.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:15.043 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:05:15.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:15.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:15.402 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/52610,v1:127.0.0.1:6819/52610] [v2:127.0.0.1:6820/52610,v1:127.0.0.1:6821/52610] exists,up 9f1bd5b2-c83e-463c-b023-d48ebd49764a 2021-05-12T12:05:15.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:15.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:15.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:15.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2021-05-12T12:05:17.506 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:17.502+0000 7f32b8e8b0c0 -1 osd.1 47 log_to_monitors {default=true} 2021-05-12T12:05:35.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2021-05-12T12:05:35.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:05:35.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:05:35.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:35.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:35.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:35.407 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:05:35.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2021-05-12T12:05:35.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok dump_ops_in_flight 2021-05-12T12:05:35.506 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:05:35.506 INFO:tasks.workunit.client.0.smithi090.stdout: "ops": [], 2021-05-12T12:05:35.506 INFO:tasks.workunit.client.0.smithi090.stdout: "num_ops": 0 2021-05-12T12:05:35.507 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:05:35.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2021-05-12T12:05:35.515 INFO:tasks.workunit.client.0.smithi090.stdout:allowing recovery 2021-05-12T12:05:35.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:35.857 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:35.858 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T12:05:30.982798+0000 47'102 52:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:05:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:05:35.858 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:35.859 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:05:35.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:35.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:05:35.992 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:36.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:36.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:05:36.125 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:36.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T12:05:36.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:05:36.257 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:05:36.265 INFO:tasks.workunit.client.0.smithi090.stdout:reading divergent objects 2021-05-12T12:05:36.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2021-05-12T12:05:36.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2021-05-12T12:05:36.603 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:05:36.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T12:05:30.982798+0000 47'102 52:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:02:48.854657+0000 0'0 2021-05-12T12:02:48.854657+0000 0 2021-05-12T12:05:36.604 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:05:36.604 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:05:36.605 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:05:36.614 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2021-05-12T12:05:36.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2021-05-12T12:05:36.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:36.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:05:36.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:36.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:05:36.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:36.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:05:36.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:05:36.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:05:36.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2021-05-12T12:05:36.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:05:36.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T12:05:36.786 INFO:tasks.workunit.client.0.smithi090.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T12:05:36.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:05:36.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T12:05:36.816 INFO:tasks.workunit.client.0.smithi090.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T12:05:36.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2021-05-12T12:05:36.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:36.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:05:36.823 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4547.0:1 2021-05-12T12:03:16.588172+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:36.824 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4547.0:1 2021-05-12T12:03:16.588172+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:36.824 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:36.824 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2021-05-12T12:05:36.824 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: more recent entry found: 41'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4660.0:1 2021-05-12T12:04:13.888359+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T12:05:36.825 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2021-05-12T12:03:16.583269+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.825 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4542.0:1 2021-05-12T12:03:16.583269+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.825 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:36.825 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2021-05-12T12:05:36.826 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T12:05:36.826 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T12:05:36.826 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T12:03:16.586884+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:36.826 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T12:03:16.586884+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:36.827 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T12:05:36.827 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2021-05-12T12:05:36.827 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T12:05:36.827 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:36.827 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2021-05-12T12:05:36.828 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T12:03:16.586884+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:05:36.828 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T12:05:36.828 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2021-05-12T12:05:36.828 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2021-05-12T12:03:16.569076+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.828 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2021-05-12T12:03:16.569076+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.829 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:36.829 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'101 last_divergent_update: 28'101 2021-05-12T12:05:36.829 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T12:05:36.829 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:36.830 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2021-05-12T12:05:36.830 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'101 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2021-05-12T12:03:16.569076+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.830 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T12:05:36.830 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2021-05-12T12:05:36.830 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4535.0:1 2021-05-12T12:03:16.578440+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.831 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4535.0:1 2021-05-12T12:03:16.578440+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.831 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:36.831 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'102 last_divergent_update: 28'102 2021-05-12T12:05:36.831 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T12:05:36.832 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:05:36.832 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2021-05-12T12:05:36.832 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'102 (28'1) modify 1:d858c035:::existing_1:head by client.4535.0:1 2021-05-12T12:03:16.578440+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.832 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T12:05:36.833 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2021-05-12T12:05:36.833 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4541.0:1 2021-05-12T12:03:16.582750+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.833 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4541.0:1 2021-05-12T12:03:16.582750+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:05:36.833 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:05:36.833 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2021-05-12T12:05:36.834 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T12:05:36.834 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:04:57.632+0000 7fbe0c60f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T12:05:36.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:36.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:05:36.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T12:05:36.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:05:36.836 INFO:tasks.workunit.client.0.smithi090.stdout:success 2021-05-12T12:05:36.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2021-05-12T12:05:36.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.29659.out 2021-05-12T12:05:36.837 INFO:tasks.workunit.client.0.smithi090.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.29659.out': No such file or directory 2021-05-12T12:05:36.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2021-05-12T12:05:36.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:05:36.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:05:37.355 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' does not exist 2021-05-12T12:05:37.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2021-05-12T12:05:37.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:37.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:37.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:37.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:37.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:37.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:05:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:05:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:05:37.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:05:37.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:37.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:37.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:37.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:37.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:37.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:37.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:05:37.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:05:37.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:05:37.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:05:37.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:05:37.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:05:37.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:37.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:05:37.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:05:37.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:37.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:05:37.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:05:37.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:05:37.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:05:37.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:37.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:37.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:05:37.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:05:37.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:05:37.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:05:37.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:05:37.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:05:37.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:05:37.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:05:37.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:05:37.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:05:37.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:05:37.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:05:37.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:05:37.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:05:37.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:05:37.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:05:37.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:05:37.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:05:37.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:05:37.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:05:37.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:05:37.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:05:37.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:37.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:05:37.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:05:37.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:05:37.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:05:37.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:05:37.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:05:37.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:05:37.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:37.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:37.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:05:37.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:05:37.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:05:37.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:05:37.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:05:37.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:37.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:37.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29659 2021-05-12T12:05:37.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:05:37.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:05:37.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:05:37.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:05:37.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2021-05-12T12:05:37.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2021-05-12T12:05:37.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2021-05-12T12:05:37.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2021-05-12T12:05:37.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2021-05-12T12:05:37.717 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2021-05-12T12:05:37.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2021-05-12T12:05:37.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2021-05-12T12:05:37.717 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:05:37.717 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:05:37.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2021-05-12T12:05:37.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:05:37.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:05:37.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:05:37.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:05:37.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:05:37.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:05:38.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:05:38.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:38.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:38.080 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:38.080 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:38.080 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:38.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:38.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:05:38.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:05:38.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:05:38.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:05:38.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:05:38.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:05:38.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:05:38.343 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:05:38.344 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:05:38.344 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:05:38.344 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:38.345 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:38.345 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:38.345 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:05:38.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:05:38.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get fsid 2021-05-12T12:05:38.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:05:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:05:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:05:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:05:38.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:05:38.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:05:38.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:05:38.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:05:38.439 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:05:38.440 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:38.440 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:38.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:05:38.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:05:38.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get mon_host 2021-05-12T12:05:38.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2021-05-12T12:05:38.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:05:38.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:05:38.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:05:38.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:05:38.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:05:38.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:05:39.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:05:39.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:39.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:39.026 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:39.026 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:39.026 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:39.026 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:39.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:05:39.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:05:39.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:39.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2021-05-12T12:05:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:05:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:39.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:05:39.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:39.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:39.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:39.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:39.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:05:39.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:05:39.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:39.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:39.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:39.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:39.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:39.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:39.063 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:39.063 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:39.063 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:39.063 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:39.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:39.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:39.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:39.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:39.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:39.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:39.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:39.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:39.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:39.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:39.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:39.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:05:39.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:39.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:05:39.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 caff8b80-0f3f-4420-bed0-4c96fe2cdd1e' 2021-05-12T12:05:39.068 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:05:39.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:39.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCTxJtgSqpdBBAAHAyP3PwThB8XSfVLarYF4g== 2021-05-12T12:05:39.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCTxJtgSqpdBBAAHAyP3PwThB8XSfVLarYF4g=="}' 2021-05-12T12:05:39.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new caff8b80-0f3f-4420-bed0-4c96fe2cdd1e -i td/divergent-priors/0/new.json 2021-05-12T12:05:39.388 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:05:39.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:05:39.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCTxJtgSqpdBBAAHAyP3PwThB8XSfVLarYF4g== --osd-uuid caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:05:39.429 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.425+0000 7f40f3cec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:39.431 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.427+0000 7f40f3cec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:39.432 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.428+0000 7f40f3cec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:39.433 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.429+0000 7f40f3cec0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:39.433 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.429+0000 7f40f3cec0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:39.433 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.429+0000 7f40f3cec0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:05:39.472 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:39.469+0000 7f40f3cec0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:05:44.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:05:44.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:05:44.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:05:44.186 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:05:44.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:05:44.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:05:44.671 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:05:44.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:44.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:05:44.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:05:44.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:05:44.699 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:44.695+0000 7f9f094a10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:44.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:44.697+0000 7f9f094a10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:44.702 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:44.699+0000 7f9f094a10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:45.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:05:45.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:45.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:05:45.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:45.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:45.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:45.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:45.038 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:05:45.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:45.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:45.225 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:45.222+0000 7f9f094a10c0 -1 Falling back to public interface 2021-05-12T12:05:45.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:46.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:46.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:46.425 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:05:46.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:05:46.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:46.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:46.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:47.439 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:47.436+0000 7f9f094a10c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:05:47.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:47.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:47.797 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:05:47.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:05:47.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:47.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:05:48.173 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56090,v1:127.0.0.1:6803/56090] [v2:127.0.0.1:6804/56090,v1:127.0.0.1:6805/56090] exists,up caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:05:48.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:48.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:48.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:48.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:48.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2021-05-12T12:05:48.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:48.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:48.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:05:48.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:48.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:05:48.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:48.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:48.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:48.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:48.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:05:48.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:05:48.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:48.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:48.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:48.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:48.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:48.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:48.182 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:48.182 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:48.182 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:48.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:48.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:48.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:48.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:48.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:48.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:48.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:48.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:48.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:48.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:48.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:48.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:48.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:05:48.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:48.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:05:48.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 7f44dd8d-eca8-43af-a678-32c354a5a07d' 2021-05-12T12:05:48.186 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:05:48.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:48.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCcxJtgPJWHCxAAcYf/2XD6AnWvpZekmVxdtg== 2021-05-12T12:05:48.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCcxJtgPJWHCxAAcYf/2XD6AnWvpZekmVxdtg=="}' 2021-05-12T12:05:48.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7f44dd8d-eca8-43af-a678-32c354a5a07d -i td/divergent-priors/1/new.json 2021-05-12T12:05:48.713 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:05:48.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:05:48.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcxJtgPJWHCxAAcYf/2XD6AnWvpZekmVxdtg== --osd-uuid 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:05:48.755 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.751+0000 7f04c8f9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:48.756 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.753+0000 7f04c8f9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:48.758 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.754+0000 7f04c8f9c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:48.759 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.755+0000 7f04c8f9c0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:48.759 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.755+0000 7f04c8f9c0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:48.760 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.755+0000 7f04c8f9c0c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:05:48.828 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:48.824+0000 7f04c8f9c0c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:05:53.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:05:53.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:05:53.268 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:05:53.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:05:53.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:05:53.759 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:05:53.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:05:53.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:53.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:05:53.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:05:53.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:05:53.787 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:53.784+0000 7f56042d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:53.800 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:53.797+0000 7f56042d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:53.802 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:53.798+0000 7f56042d30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:05:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:05:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:05:54.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:05:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:05:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:54.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:05:54.126 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:05:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:54.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:54.320 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:54.316+0000 7f56042d30c0 -1 Falling back to public interface 2021-05-12T12:05:54.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:55.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:55.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:55.512 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:05:55.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:05:55.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:55.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:55.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:05:56.546 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:56.542+0000 7f56042d30c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:05:56.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:05:56.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:05:56.927 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:05:56.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:05:56.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:05:56.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:05:57.296 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57475,v1:127.0.0.1:6811/57475] [v2:127.0.0.1:6812/57475,v1:127.0.0.1:6813/57475] exists,up 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:05:57.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:05:57.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:05:57.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:05:57.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T12:05:57.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2021-05-12T12:05:57.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:05:57.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:05:57.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:05:57.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:05:57.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:05:57.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:05:57.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:05:57.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:05:57.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:05:57.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:05:57.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:05:57.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:05:57.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:05:57.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:05:57.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:05:57.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:05:57.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:05:57.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:05:57.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:05:57.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:05:57.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:57.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:05:57.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:05:57.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:05:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:05:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:05:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:05:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:05:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:05:57.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:05:57.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:05:57.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:05:57.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:05:57.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:05:57.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:05:57.310 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:05:57.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 9fb1ffaa-ed93-42e3-9c56-62b59ef90144' 2021-05-12T12:05:57.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:05:57.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQClxJtgBkraEhAAceOTjcox86sdPHvsTJ1AEg== 2021-05-12T12:05:57.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQClxJtgBkraEhAAceOTjcox86sdPHvsTJ1AEg=="}' 2021-05-12T12:05:57.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 -i td/divergent-priors/2/new.json 2021-05-12T12:05:57.872 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:05:57.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:05:57.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQClxJtgBkraEhAAceOTjcox86sdPHvsTJ1AEg== --osd-uuid 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:05:57.914 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.911+0000 7fd25a5c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:57.915 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.913+0000 7fd25a5c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:57.917 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.914+0000 7fd25a5c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:05:57.918 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.915+0000 7fd25a5c60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:57.918 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.915+0000 7fd25a5c60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:57.918 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.915+0000 7fd25a5c60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:05:57.956 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:05:57.953+0000 7fd25a5c60c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:06:02.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:06:02.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:06:02.551 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:06:02.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:06:02.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:06:03.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:06:03.042 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:06:03.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:03.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:06:03.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:06:03.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:06:03.069 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:03.067+0000 7fe1bfda30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:03.072 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:03.069+0000 7fe1bfda30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:03.073 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:03.070+0000 7fe1bfda30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:03.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:06:03.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:06:03.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:06:03.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:06:03.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:06:03.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:03.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:06:03.409 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:06:03.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:03.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:03.587 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:03.585+0000 7fe1bfda30c0 -1 Falling back to public interface 2021-05-12T12:06:03.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:04.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:04.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:04.816 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:06:04.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:06:04.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:04.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:05.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:05.802 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:05.799+0000 7fe1bfda30c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:06:06.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:06.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:06.199 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:06:06.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:06:06.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:06.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:06.690 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58868,v1:127.0.0.1:6819/58868] [v2:127.0.0.1:6820/58868,v1:127.0.0.1:6821/58868] exists,up 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:06:06.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:06:06.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:06:06.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:06:06.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2021-05-12T12:06:07.499 INFO:tasks.workunit.client.0.smithi090.stderr:noout is set 2021-05-12T12:06:07.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2021-05-12T12:06:08.192 INFO:tasks.workunit.client.0.smithi090.stderr:noin is set 2021-05-12T12:06:08.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2021-05-12T12:06:08.903 INFO:tasks.workunit.client.0.smithi090.stderr:nodown is set 2021-05-12T12:06:08.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2021-05-12T12:06:08.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T12:06:09.648 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' already exists 2021-05-12T12:06:09.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:06:10.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2021-05-12T12:06:11.321 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 3 2021-05-12T12:06:11.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2021-05-12T12:06:12.058 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 min_size to 2 2021-05-12T12:06:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2021-05-12T12:06:12.693 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 pg_autoscale_mode to on 2021-05-12T12:06:12.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2021-05-12T12:06:12.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:06:12.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:06:13.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:06:13.066 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:06:13.066 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:06:13.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:06:13.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:13.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:06:13.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T12:06:13.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T12:06:13.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T12:06:13.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:13.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:06:13.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T12:06:13.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T12:06:13.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-05-12T12:06:13.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:13.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:06:13.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T12:06:13.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T12:06:13.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509444' 2021-05-12T12:06:13.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:13.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T12:06:13.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:13.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:06:13.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T12:06:13.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:13.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T12:06:13.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T12:06:13.473 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836487 2021-05-12T12:06:13.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:13.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836487 2021-05-12T12:06:13.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:13.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T12:06:13.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:13.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:06:13.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T12:06:13.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:13.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T12:06:13.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T12:06:13.839 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672966 2021-05-12T12:06:13.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:06:14.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T12:06:14.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:14.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T12:06:14.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:14.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:06:14.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T12:06:14.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:14.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T12:06:14.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T12:06:14.233 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509444 2021-05-12T12:06:14.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:14.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T12:06:14.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2021-05-12T12:06:14.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:06:14.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:06:14.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:06:14.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:06:14.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:06:14.602 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:06:14.602 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:06:14.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:06:14.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:06:14.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:06:14.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:06:14.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:06:14.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:06:14.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:06:14.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:06:15.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:06:15.050 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:06:15.050 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:06:15.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:06:15.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:15.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:06:15.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T12:06:15.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T12:06:15.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T12:06:15.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:15.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:06:15.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T12:06:15.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T12:06:15.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-05-12T12:06:15.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:15.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:06:15.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509446 2021-05-12T12:06:15.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509446 2021-05-12T12:06:15.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509446' 2021-05-12T12:06:15.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:15.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T12:06:15.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:15.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:06:15.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T12:06:15.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:15.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T12:06:15.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T12:06:15.452 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836490 2021-05-12T12:06:15.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:15.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T12:06:15.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:15.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T12:06:15.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:15.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:06:15.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T12:06:15.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:15.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T12:06:15.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T12:06:15.819 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672968 2021-05-12T12:06:15.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:06:16.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672968 -lt 42949672968 2021-05-12T12:06:16.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:16.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-05-12T12:06:16.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:16.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:06:16.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509446 2021-05-12T12:06:16.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:16.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509446 2021-05-12T12:06:16.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-05-12T12:06:16.197 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509446 2021-05-12T12:06:16.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:16.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509446 -lt 64424509446 2021-05-12T12:06:16.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:06:16.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:16.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:17.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:06:17.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:17.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:17.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:17.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:17.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:17.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:17.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:17.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:06:17.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:17.407 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:17.407 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:06:17.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:06:17.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:06:17.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2021-05-12T12:06:17.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2021-05-12T12:06:18.231 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:06:18.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2021-05-12T12:06:18.241 INFO:tasks.workunit.client.0.smithi090.stdout:primary and soon to be divergent is 1 2021-05-12T12:06:18.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2021-05-12T12:06:18.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:18.579 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:18.580 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:06:10.245006+0000 0'0 28:11 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:06:18.580 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:06:18.580 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:18.581 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:06:18.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2021-05-12T12:06:18.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:06:18.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2021-05-12T12:06:18.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2021-05-12T12:06:18.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:06:18.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2021-05-12T12:06:18.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2021-05-12T12:06:18.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T12:06:18.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2021-05-12T12:06:18.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2021-05-12T12:06:18.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2021-05-12T12:06:18.594 INFO:tasks.workunit.client.0.smithi090.stdout:writing initial objects 2021-05-12T12:06:18.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2021-05-12T12:06:18.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:18.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2021-05-12T12:06:18.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:18.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2021-05-12T12:06:18.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:18.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2021-05-12T12:06:18.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:18.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2021-05-12T12:06:19.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2021-05-12T12:06:19.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2021-05-12T12:06:19.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2021-05-12T12:06:19.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2021-05-12T12:06:19.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2021-05-12T12:06:19.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2021-05-12T12:06:19.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2021-05-12T12:06:19.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:19.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2021-05-12T12:06:20.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2021-05-12T12:06:20.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2021-05-12T12:06:20.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2021-05-12T12:06:20.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2021-05-12T12:06:20.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2021-05-12T12:06:20.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2021-05-12T12:06:20.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2021-05-12T12:06:20.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:20.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2021-05-12T12:06:21.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2021-05-12T12:06:21.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2021-05-12T12:06:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2021-05-12T12:06:21.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2021-05-12T12:06:21.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2021-05-12T12:06:21.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2021-05-12T12:06:21.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2021-05-12T12:06:21.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2021-05-12T12:06:21.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:21.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2021-05-12T12:06:22.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2021-05-12T12:06:22.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2021-05-12T12:06:22.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2021-05-12T12:06:22.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2021-05-12T12:06:22.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2021-05-12T12:06:22.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2021-05-12T12:06:22.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2021-05-12T12:06:22.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:22.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2021-05-12T12:06:23.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2021-05-12T12:06:23.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2021-05-12T12:06:23.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2021-05-12T12:06:23.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2021-05-12T12:06:23.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2021-05-12T12:06:23.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2021-05-12T12:06:23.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:23.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2021-05-12T12:06:24.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2021-05-12T12:06:24.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2021-05-12T12:06:24.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2021-05-12T12:06:24.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2021-05-12T12:06:24.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2021-05-12T12:06:24.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2021-05-12T12:06:24.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2021-05-12T12:06:24.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:24.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2021-05-12T12:06:25.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2021-05-12T12:06:25.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2021-05-12T12:06:25.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2021-05-12T12:06:25.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2021-05-12T12:06:25.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2021-05-12T12:06:25.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2021-05-12T12:06:25.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2021-05-12T12:06:25.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:25.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2021-05-12T12:06:26.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2021-05-12T12:06:26.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2021-05-12T12:06:26.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2021-05-12T12:06:26.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2021-05-12T12:06:26.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2021-05-12T12:06:26.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2021-05-12T12:06:26.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:26.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2021-05-12T12:06:27.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2021-05-12T12:06:27.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2021-05-12T12:06:27.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2021-05-12T12:06:27.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2021-05-12T12:06:27.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2021-05-12T12:06:27.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2021-05-12T12:06:27.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2021-05-12T12:06:27.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:27.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2021-05-12T12:06:28.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2021-05-12T12:06:28.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2021-05-12T12:06:28.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2021-05-12T12:06:28.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2021-05-12T12:06:28.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2021-05-12T12:06:28.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2021-05-12T12:06:28.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2021-05-12T12:06:28.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:28.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2021-05-12T12:06:29.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2021-05-12T12:06:29.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2021-05-12T12:06:29.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2021-05-12T12:06:29.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2021-05-12T12:06:29.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2021-05-12T12:06:29.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2021-05-12T12:06:29.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:29.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2021-05-12T12:06:30.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2021-05-12T12:06:30.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2021-05-12T12:06:30.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2021-05-12T12:06:30.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2021-05-12T12:06:30.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2021-05-12T12:06:30.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2021-05-12T12:06:30.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2021-05-12T12:06:30.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:30.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2021-05-12T12:06:31.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T12:06:31.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2021-05-12T12:06:31.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:06:31.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2021-05-12T12:06:31.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:06:31.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:06:31.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:06:31.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:06:31.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:06:31.160 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:06:31.160 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:06:31.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:06:31.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:06:31.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:06:31.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:06:31.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:06:31.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:06:31.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:06:31.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:06:31.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:06:31.580 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:06:31.580 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:06:31.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:06:31.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:31.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:06:31.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836495 2021-05-12T12:06:31.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836495 2021-05-12T12:06:31.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836495' 2021-05-12T12:06:31.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:31.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:06:31.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672973 2021-05-12T12:06:31.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672973 2021-05-12T12:06:31.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836495 1-42949672973' 2021-05-12T12:06:31.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:06:31.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:06:31.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509451 2021-05-12T12:06:31.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509451 2021-05-12T12:06:31.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509451' 2021-05-12T12:06:31.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:31.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-05-12T12:06:31.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:31.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:06:31.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836495 2021-05-12T12:06:31.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:31.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836495 2021-05-12T12:06:31.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-05-12T12:06:31.999 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836495 2021-05-12T12:06:31.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:32.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836493 -lt 21474836495 2021-05-12T12:06:32.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:06:33.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:06:33.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:06:33.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836495 -lt 21474836495 2021-05-12T12:06:33.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:33.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2021-05-12T12:06:33.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:33.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:06:33.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672973 2021-05-12T12:06:33.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:33.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672973 2021-05-12T12:06:33.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-05-12T12:06:33.766 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672973 2021-05-12T12:06:33.766 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:06:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672973 2021-05-12T12:06:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:06:34.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509451 2021-05-12T12:06:34.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:06:34.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:06:34.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509451 2021-05-12T12:06:34.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:06:34.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509451 2021-05-12T12:06:34.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-05-12T12:06:34.134 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509451 2021-05-12T12:06:34.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:06:34.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509451 2021-05-12T12:06:34.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:06:34.495 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:34.495 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:34.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:06:34.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:06:34.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:06:34.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:06:34.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:06:34.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:06:34.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:06:34.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:06:35.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:06:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:06:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:06:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:06:35.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:06:35.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:06:35.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:06:35.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2021-05-12T12:06:35.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:06:35.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:06:35.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:06:35.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:06:36.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2021-05-12T12:06:36.195 INFO:tasks.workunit.client.0.smithi090.stdout:blackholing osds 0 2 2021-05-12T12:06:36.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2021-05-12T12:06:36.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:36.558 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:36.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:06:10.245006+0000 31'100 31:111 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:06:36.558 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:06:36.558 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:36.559 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:06:36.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:36.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2021-05-12T12:06:36.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:06:36.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:06:36.571 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:06:36.571 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:36.571 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:06:36.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:06:36.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:06:36.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:06:36.660 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:06:36.660 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:06:36.660 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:06:36.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:36.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2021-05-12T12:06:36.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:06:36.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:06:36.670 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:06:36.670 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:36.670 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:06:36.670 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:06:36.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:06:36.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:06:36.759 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:06:36.759 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:06:36.760 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:06:36.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2021-05-12T12:06:36.768 INFO:tasks.workunit.client.0.smithi090.stdout:create new divergent objects 2021-05-12T12:06:36.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2021-05-12T12:06:36.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:06:36.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:06:36.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2021-05-12T12:06:36.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2021-05-12T12:06:36.770 INFO:tasks.workunit.client.0.smithi090.stdout:writing divergent objects 2021-05-12T12:06:36.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2021-05-12T12:06:36.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2021-05-12T12:06:36.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:06:36.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T12:06:36.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T12:06:36.772 INFO:tasks.workunit.client.0.smithi090.stdout:remove divergent objects 2021-05-12T12:06:36.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2021-05-12T12:06:36.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T12:06:36.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2021-05-12T12:06:36.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:06:36.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2021-05-12T12:06:36.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2021-05-12T12:06:36.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T12:06:36.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2021-05-12T12:06:36.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2021-05-12T12:06:36.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2021-05-12T12:06:36.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2021-05-12T12:06:36.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2021-05-12T12:06:46.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2021-05-12T12:06:46.779 INFO:tasks.workunit.client.0.smithi090.stderr:rados: no process found 2021-05-12T12:06:46.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2021-05-12T12:06:46.780 INFO:tasks.workunit.client.0.smithi090.stdout:killing all the osds 2021-05-12T12:06:46.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:47.136 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:47.136 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.137 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.137 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.137 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.137 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.137 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.138 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.138 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.138 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.138 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.139 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.139 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.139 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.139 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.140 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.140 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.140 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.140 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.140 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.141 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.141 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.141 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.141 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.141 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.142 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.142 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.142 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.142 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.143 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.143 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.143 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.143 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.143 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.144 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.144 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.144 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.144 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.145 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.145 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.145 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.145 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.145 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.146 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.146 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.146 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.146 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.146 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.147 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.147 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.147 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.147 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.149 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.149 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.149 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.149 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.150 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.150 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.150 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.150 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.150 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.151 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.151 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.151 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.151 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.151 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.152 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.152 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.152 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.152 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.153 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2021-05-12T12:06:10.245006+0000 31'106 31:117 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:06:47.153 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.153 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.153 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.153 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.154 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.154 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.154 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.154 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.155 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.155 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.155 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.155 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.155 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.156 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.156 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.156 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.156 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.157 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.157 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.157 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.157 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.157 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.158 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.158 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.158 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.158 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.159 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.159 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.159 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.159 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.159 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.160 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.160 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.160 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.160 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.161 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.161 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.161 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.161 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.161 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.162 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.162 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.162 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.162 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.163 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.163 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.163 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.163 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.163 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.164 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.164 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.164 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.164 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:47.165 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:06:47.165 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:47.166 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:06:47.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:06:47.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:06:47.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:06:47.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:06:47.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:06:47.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:06:47.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:06:47.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:47.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2021-05-12T12:06:47.757 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:06:47.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:47.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2021-05-12T12:06:48.275 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:06:48.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T12:06:48.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2021-05-12T12:06:48.816 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:06:48.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:48.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2021-05-12T12:06:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already out. 2021-05-12T12:06:49.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:49.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2021-05-12T12:06:49.859 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already out. 2021-05-12T12:06:49.868 INFO:tasks.workunit.client.0.smithi090.stdout:bringing up non_divergent 0 2 2021-05-12T12:06:49.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2021-05-12T12:06:49.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:06:50.215 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:06:50.216 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.216 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.216 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.216 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.217 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.217 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.217 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.217 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.217 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.218 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.218 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.218 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.218 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.218 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.219 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.219 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.219 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.219 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.220 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.220 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.220 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.220 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.220 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.221 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.221 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.221 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.221 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.221 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.222 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.222 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.222 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.222 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.223 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.223 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.223 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.223 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.223 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.224 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.224 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.224 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.224 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.224 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.225 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.225 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.225 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.225 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.226 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.226 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.226 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.226 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.227 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.227 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.227 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.227 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.227 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.228 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.228 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.228 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.228 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.228 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.229 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.229 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.229 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.229 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.229 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.230 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.230 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.230 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.230 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.231 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.231 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.231 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.231 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.231 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.232 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2021-05-12T12:06:10.245006+0000 31'106 31:117 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:06:50.232 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.232 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.232 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.233 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.233 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.233 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.233 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.233 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.234 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.234 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.234 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.234 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.235 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.235 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.235 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.235 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.235 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.236 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.236 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.236 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.236 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.236 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.237 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.237 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.237 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.237 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.238 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.238 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.238 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.238 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.238 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.239 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.239 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.239 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.239 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.240 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.240 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.240 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.240 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.240 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.241 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.241 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.241 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.241 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 0 0 0 0 0 0 0 0 0 0 unknown 2021-05-12T12:06:45.656414+0000 0'0 0:0 [] -1 [] -1 0'0 2021-05-12T12:06:45.656414+0000 0'0 2021-05-12T12:06:45.656414+0000 0 2021-05-12T12:06:50.244 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:06:50.244 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:06:50.247 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:06:50.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:50.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2021-05-12T12:06:50.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:06:50.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:06:50.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:06:50.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:06:50.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:06:50.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:06:50.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:06:50.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:06:50.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:06:50.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:06:50.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:06:50.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:06:50.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:06:50.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:06:50.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:06:50.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:06:50.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:06:50.251 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:06:50.252 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:50.252 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:06:50.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:06:50.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:06:50.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:06:50.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:06:50.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:06:50.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:06:50.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:06:50.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:50.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:06:50.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:06:50.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:06:50.255 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:06:50.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:06:50.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:50.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:06:50.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:06:50.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:06:50.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:06:50.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:06:50.258 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:50.254+0000 7f2f158f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:50.259 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:50.255+0000 7f2f158f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:50.260 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:50.257+0000 7f2f158f20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:50.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:06:50.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:06:50.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:06:50.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:06:50.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:06:50.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:50.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:06:50.597 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:06:50.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:50.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:50.775 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:50.771+0000 7f2f158f20c0 -1 Falling back to public interface 2021-05-12T12:06:50.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:51.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:51.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:51.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:06:51.962 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:06:51.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:51.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:52.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:53.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:53.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:53.336 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:06:53.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:06:53.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:53.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:53.464 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:53.461+0000 7f2f158f20c0 -1 osd.0 31 log_to_monitors {default=true} 2021-05-12T12:06:53.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:54.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:54.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:54.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:06:54.738 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:06:54.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:54.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:06:55.099 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up out weight 0 up_from 41 up_thru 0 down_at 35 last_clean_interval [5,31) [v2:127.0.0.1:6802/62927,v1:127.0.0.1:6803/62927] [v2:127.0.0.1:6804/62927,v1:127.0.0.1:6805/62927] exists,up caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:06:55.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:06:55.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:06:55.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:06:55.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-05-12T12:06:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:06:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:06:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:06:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:06:55.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:06:55.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:06:55.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:06:55.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:06:55.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:06:55.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:06:55.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:06:55.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:06:55.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:06:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:06:55.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:06:55.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:06:55.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:06:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:06:55.109 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:06:55.109 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:06:55.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:06:55.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:06:55.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:06:55.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:06:55.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:06:55.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:06:55.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:06:55.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:55.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:06:55.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:06:55.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:06:55.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:06:55.112 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:06:55.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:06:55.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:06:55.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:06:55.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:06:55.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:06:55.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:06:55.130 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:55.127+0000 7f4c9e2cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:55.132 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:55.129+0000 7f4c9e2cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:55.134 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:55.130+0000 7f4c9e2cb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:06:55.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:06:55.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:06:55.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:06:55.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:06:55.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:06:55.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:55.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:06:55.472 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:06:55.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:55.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:55.653 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:55.650+0000 7f4c9e2cb0c0 -1 Falling back to public interface 2021-05-12T12:06:55.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:56.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:56.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:56.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:06:56.837 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:06:56.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:56.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:57.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:06:58.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:06:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:06:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:06:58.233 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:06:58.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:06:58.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:06:58.374 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:06:58.372+0000 7f4c9e2cb0c0 -1 osd.2 31 log_to_monitors {default=true} 2021-05-12T12:06:58.689 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up out weight 0 up_from 43 up_thru 0 down_at 37 last_clean_interval [15,31) [v2:127.0.0.1:6810/63518,v1:127.0.0.1:6811/63518] [v2:127.0.0.1:6812/63518,v1:127.0.0.1:6813/63518] exists,up 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:06:58.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:06:58.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:06:58.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:06:58.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:58.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2021-05-12T12:06:59.217 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already in. 2021-05-12T12:06:59.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:06:59.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2021-05-12T12:06:59.940 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already in. 2021-05-12T12:06:59.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:06:59.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2021-05-12T12:06:59.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:06:59.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:06:59.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:06:59.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:06:59.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:06:59.953 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:06:59.953 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:06:59.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:06:59.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:06:59.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:07:00.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:07:00.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:07:00.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:07:00.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:07:00.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:07:00.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:07:00.485 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:07:00.486 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:07:00.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:07:00.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:00.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:07:00.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659140 2021-05-12T12:07:00.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659140 2021-05-12T12:07:00.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659140' 2021-05-12T12:07:00.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:00.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:07:00.731 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:07:00.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:07:00.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:07:00.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:07:00.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:00.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:07:00.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593731 2021-05-12T12:07:00.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593731 2021-05-12T12:07:00.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659140 2-184683593731' 2021-05-12T12:07:00.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:00.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-176093659140 2021-05-12T12:07:00.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:00.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:07:00.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-176093659140 2021-05-12T12:07:00.875 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:00.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659140 2021-05-12T12:07:00.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2021-05-12T12:07:00.876 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 176093659140 2021-05-12T12:07:00.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:01.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659138 -lt 176093659140 2021-05-12T12:07:01.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:07:02.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:07:02.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:02.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659140 -lt 176093659140 2021-05-12T12:07:02.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:02.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-184683593731 2021-05-12T12:07:02.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:02.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:07:02.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-184683593731 2021-05-12T12:07:02.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:02.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593731 2021-05-12T12:07:02.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2021-05-12T12:07:02.799 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 184683593731 2021-05-12T12:07:02.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:07:03.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593731 -lt 184683593731 2021-05-12T12:07:03.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:07:03.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:03.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:03.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 128 == 0 2021-05-12T12:07:03.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:03.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:03.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:03.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:03.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:03.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:03.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:04.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:04.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:04.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:04.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:04.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:04.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:07:04.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:07:04.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:07:04.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:04.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:07:04.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:04.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:04.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:04.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:04.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:04.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:04.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:04.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:04.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:04.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:04.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:04.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:05.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:05.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:05.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:05.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:05.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:05.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:05.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:05.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:05.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:05.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:05.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:05.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:07:05.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:05.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:07:06.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:06.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:06.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:06.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:06.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:06.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:06.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:06.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:06.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:06.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:06.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:06.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:07.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:07.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:07.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:07.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:07.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:07.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:07.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:07.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:07.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:07.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:07.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:07.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:07:07.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:07.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:07:07.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:07.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:07.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:07.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:07.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:07.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:07.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:07.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:08.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:08.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:08.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:08.258 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:08.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:08.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:08.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:08.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:08.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:08.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:08.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:08.738 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:08.738 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:09.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:09.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:09.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:07:09.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:09.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:07:10.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:10.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:10.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:10.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:10.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:10.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:10.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:10.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:10.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:10.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:10.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:10.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:10.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:10.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:10.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:10.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:10.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:10.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:10.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:10.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:10.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:11.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:11.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:11.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:07:11.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:11.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:07:12.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:12.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:12.975 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:12.975 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:12.975 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:12.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:12.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:12.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:13.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:13.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:13.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:13.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:13.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:13.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:13.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:13.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:13.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:13.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:13.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:13.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:13.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:14.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:14.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:14.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:07:17.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:17.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:17.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:17.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:17.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:17.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:17.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:17.530 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:17.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:17.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:17.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:17.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:18.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:18.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:18.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:18.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:18.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:18.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:18.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:18.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:18.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:18.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:18.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:18.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:07:18.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:18.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:07:25.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:25.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:25.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:25.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:25.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:25.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:25.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:25.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:25.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:25.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:25.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:25.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:26.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:26.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:26.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:26.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:26.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:26.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:26.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:26.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:26.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:26.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:26.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:26.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:07:26.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:26.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:07:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:33.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:33.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:33.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:33.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:33.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:33.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:34.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:34.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:34.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:34.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:34.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:34.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:34.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:35.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:35.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:35.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:07:35.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:07:35.728 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:07:35.729 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "ef34446d-c6a9-4dba-bad5-4b4f94d7bb47", 2021-05-12T12:07:35.729 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:35.729 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:07:35.729 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:07:35.725147+0000", 2021-05-12T12:07:35.730 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:07:35.730 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:07:35.730 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:07:35.730 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:07:35.731 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:07:35.731 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:07:35.731 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:07:35.731 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:07:35.731 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:07:35.732 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.732 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:07:35.732 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.732 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:07:35.733 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.733 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.733 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:07:35.733 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.734 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:07:35.734 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:35.734 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:07:35.734 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:07:35.734 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:07:35.735 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.735 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:07:35.735 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:07:35.735 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.735 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:07:35.736 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:35.736 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:07:35.736 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:07:35.736 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:07:35.737 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.737 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:07:35.737 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.737 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:07:35.737 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.738 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.738 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:07:35.738 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.738 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:07:35.739 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:35.739 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:07:35.739 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 100 pgs degraded", 2021-05-12T12:07:35.739 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 100 2021-05-12T12:07:35.739 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.740 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:07:35.740 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.740 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.740 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.741 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.741 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.741 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.741 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.741 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.742 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.742 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.742 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.742 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.743 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.743 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.743 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.743 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.743 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.744 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.744 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.744 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.744 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.744 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.745 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.745 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.745 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.745 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.746 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.746 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.746 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.746 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.747 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.747 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.747 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.747 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.747 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.748 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.748 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.748 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.748 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.749 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.749 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.749 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.749 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.749 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.750 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.750 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.750 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.750 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.751 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.751 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.751 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.751 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.751 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.752 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.752 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.752 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.752 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.752 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.753 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.753 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.753 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.753 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.754 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.754 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.754 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.754 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.755 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.755 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.755 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.755 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.755 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.756 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.756 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.756 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.20 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.756 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.757 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.757 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.21 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.757 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.757 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.757 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.22 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.758 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.758 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.758 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.23 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.758 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.759 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.759 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.4e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.759 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.759 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.759 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.4f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.760 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.760 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.760 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.50 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.760 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.761 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.761 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.51 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.761 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.761 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.761 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.52 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.762 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.762 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.762 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.53 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.762 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.763 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.763 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.54 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.763 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.764 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.764 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.55 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.764 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.764 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.764 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.56 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.765 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.765 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.765 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.57 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.765 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.766 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.766 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.58 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.766 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.766 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.766 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.59 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.767 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.767 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.767 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5a is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.767 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.768 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.768 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5b is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.768 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.768 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.768 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5c is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.769 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.769 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.769 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5d is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.769 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.770 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.770 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5e is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.770 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.770 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.770 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5f is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.771 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.771 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.771 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.60 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.771 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.772 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.772 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.61 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.772 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.772 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.773 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.62 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.773 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.773 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.773 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.63 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.773 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.774 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.774 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.64 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:07:35.774 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.774 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.774 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:07:35.775 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.775 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:07:35.775 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:07:35.775 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:07:35.775 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:07:35.776 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:07:35.776 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.776 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:07:35.776 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.777 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:07:35.777 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.777 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.777 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:07:35.777 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.778 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.778 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:07:35.778 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.778 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.778 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:07:35.779 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.779 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:07:35.779 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:07:35.779 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:07:35.779 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:07:35.780 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:07:35.780 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:05:37.733410Z", 2021-05-12T12:07:35.780 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:05:37.733410Z", 2021-05-12T12:07:35.780 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:07:35.781 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:07:35.781 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:07:35.781 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:07:35.781 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:07:35.781 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:07:35.782 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:07:35.782 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:07:35.782 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:07:35.782 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:07:35.782 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:07:35.783 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:07:35.783 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:07:35.783 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:07:35.783 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:07:35.783 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:07:35.784 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.784 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:07:35.784 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.784 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:07:35.784 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.785 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:07:35.785 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:07:35.785 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:07:35.785 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.786 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.786 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.786 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:07:35.786 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:07:35.786 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.787 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.787 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.787 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:07:35.787 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:07:35.787 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:07:35.788 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:07:35.788 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:07:35.788 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.788 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.789 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.789 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:07:35.789 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:07:35.789 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.789 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:07:35.790 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 48, 2021-05-12T12:07:35.790 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:07:35.790 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:05:38.600554+0000", 2021-05-12T12:07:35.790 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:07:00.175434+0000", 2021-05-12T12:07:35.790 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:06:58.573450+0000", 2021-05-12T12:07:35.791 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:06:59.655908+0000", 2021-05-12T12:07:35.791 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:07:35.791 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:07:35.791 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:07:35.791 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:07:35.792 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:07:35.792 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:07:35.792 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:07:35.792 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:07:35.793 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:07:35.793 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:07:35.793 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.793 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 9, 2021-05-12T12:07:35.793 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:07:35.794 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:07:35.794 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:07:35.794 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:07:35.794 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:07:35.794 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:07:35.795 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:07:35.795 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:07:35.795 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:07:35.795 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:07:35.795 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.796 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:07:35.796 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:07:35.796 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:06:09.184020+0000", 2021-05-12T12:07:35.796 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 1, 2021-05-12T12:07:35.797 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool", 2021-05-12T12:07:35.797 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:07:35.797 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:07:35.797 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:07:35.797 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 0, 2021-05-12T12:07:35.798 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:07:35.798 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:07:35.798 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:07:35.798 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:07:35.799 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:07:35.799 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "on", 2021-05-12T12:07:35.799 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 128, 2021-05-12T12:07:35.799 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:07:35.799 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 128, 2021-05-12T12:07:35.800 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 128, 2021-05-12T12:07:35.800 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 128, 2021-05-12T12:07:35.800 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:07:35.800 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:07:35.801 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:07:35.801 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:07:35.801 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:07:35.801 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:07:35.801 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:07:35.802 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.802 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "33", 2021-05-12T12:07:35.802 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:07:35.802 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:07:35.803 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "33", 2021-05-12T12:07:35.803 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:07:35.803 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:07:35.803 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 0, 2021-05-12T12:07:35.803 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 0, 2021-05-12T12:07:35.804 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [], 2021-05-12T12:07:35.804 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:07:35.804 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:07:35.804 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:07:35.805 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:07:35.805 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:07:35.805 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:07:35.805 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:07:35.805 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:07:35.806 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:07:35.806 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:07:35.806 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:07:35.806 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:07:35.806 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:07:35.807 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:07:35.807 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:07:35.807 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "", 2021-05-12T12:07:35.807 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:07:35.808 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:07:35.808 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.808 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:07:35.808 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:07:35.808 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:07:35.809 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:07:35.809 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:07:35.809 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:07:35.809 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:07:35.809 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:07:35.810 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 0, 2021-05-12T12:07:35.810 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:07:35.810 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:07:35.810 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:07:35.811 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:07:35.811 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.811 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.811 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:07:35.811 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.812 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:07:35.812 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "caff8b80-0f3f-4420-bed0-4c96fe2cdd1e", 2021-05-12T12:07:35.812 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:07:35.812 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:07:35.812 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:07:35.813 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:07:35.813 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:07:35.813 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 31, 2021-05-12T12:07:35.813 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 41, 2021-05-12T12:07:35.814 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 47, 2021-05-12T12:07:35.814 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 35, 2021-05-12T12:07:35.814 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:07:35.815 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:07:35.815 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.815 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.815 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.815 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:07:35.816 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.816 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.816 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.816 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.817 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:07:35.817 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.817 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.817 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.818 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.818 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:07:35.818 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.818 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.819 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.819 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:07:35.819 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.819 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.820 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.820 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.820 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:07:35.820 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.820 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.821 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.821 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.821 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:35.821 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.822 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.822 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.822 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:07:35.822 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.823 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.823 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.823 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.823 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:07:35.823 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.824 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.824 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.824 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.824 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:35.824 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.825 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.825 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.825 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:07:35.825 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.826 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.826 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.826 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.826 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:07:35.826 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:07:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.827 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/62927", 2021-05-12T12:07:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/62927", 2021-05-12T12:07:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/62927", 2021-05-12T12:07:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/62927", 2021-05-12T12:07:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:07:35.828 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:07:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:07:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.829 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.830 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:07:35.830 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "7f44dd8d-eca8-43af-a678-32c354a5a07d", 2021-05-12T12:07:35.830 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:07:35.830 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:07:35.830 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:07:35.831 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:07:35.831 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:07:35.831 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:07:35.831 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 10, 2021-05-12T12:07:35.831 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 33, 2021-05-12T12:07:35.832 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 36, 2021-05-12T12:07:35.832 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:07:35.832 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:07:35.832 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.832 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.833 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.834 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:35.834 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.834 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:07:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.835 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.836 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:35.837 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:35.838 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.839 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.840 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.841 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.842 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.843 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.843 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.843 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:35.843 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:07:35.844 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.844 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.844 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.844 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/57475", 2021-05-12T12:07:35.844 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/57475", 2021-05-12T12:07:35.845 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/57475", 2021-05-12T12:07:35.845 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/57475", 2021-05-12T12:07:35.845 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:07:35.845 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:07:35.845 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.846 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.846 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.846 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:07:35.846 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "9fb1ffaa-ed93-42e3-9c56-62b59ef90144", 2021-05-12T12:07:35.846 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:07:35.847 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:07:35.847 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:07:35.847 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:07:35.847 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 15, 2021-05-12T12:07:35.848 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 31, 2021-05-12T12:07:35.848 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 43, 2021-05-12T12:07:35.848 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:07:35.848 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 37, 2021-05-12T12:07:35.848 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:07:35.849 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:07:35.849 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.849 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.849 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.850 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:07:35.850 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.850 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.850 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.851 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.851 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:07:35.851 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.851 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.852 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.852 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.852 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:07:35.852 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.852 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.853 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.853 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:07:35.853 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.853 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.854 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.854 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.854 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:07:35.854 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.854 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.855 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.855 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.855 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:07:35.855 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.856 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.856 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.856 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:07:35.856 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.856 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.857 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.857 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.857 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:07:35.857 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.857 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:07:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:07:35.858 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.859 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:07:35.859 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:07:35.859 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.859 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.860 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.860 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:07:35.860 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:07:35.860 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:07:35.860 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.861 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.861 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.861 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/63518", 2021-05-12T12:07:35.861 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/63518", 2021-05-12T12:07:35.861 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/63518", 2021-05-12T12:07:35.862 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/63518", 2021-05-12T12:07:35.862 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:07:35.862 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:07:35.862 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:07:35.862 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.863 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.863 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.863 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:07:35.863 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.863 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:07:35.864 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:47.561716+0000", 2021-05-12T12:07:35.864 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:07:35.864 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:07:35.864 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:07:35.864 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:07:35.865 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:48.458088+0000", 2021-05-12T12:07:35.865 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:07:35.865 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.865 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.865 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:07:35.866 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:48.077947+0000", 2021-05-12T12:07:35.866 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:07:35.866 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:07:35.866 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:07:35.866 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:07:35.867 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:57.533501+0000", 2021-05-12T12:07:35.867 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:07:35.867 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.867 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.868 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:07:35.868 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:48.594152+0000", 2021-05-12T12:07:35.868 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:07:35.868 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:07:35.868 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:07:35.869 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:07:35.869 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:06:06.850113+0000", 2021-05-12T12:07:35.869 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:07:35.869 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.869 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.870 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:07:35.870 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:07:35.870 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:07:35.870 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:07:35.871 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:07:35.871 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:07:35.871 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:07:35.871 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:07:35.871 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:07:35.872 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:07:35.872 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:07:35.872 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.872 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.873 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:07:35.873 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:07:35.873 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:07:35.873 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:07:35.874 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:07:35.874 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:07:35.874 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:07:35.874 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:07:35.875 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:07:35.875 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:07:35.875 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:07:35.875 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.875 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.876 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:07:35.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.876 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:07:35.876 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:07:35.876 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/62927,v1:127.0.0.1:6805/62927]", 2021-05-12T12:07:35.877 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:07:35.877 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:07:35.877 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:35.877 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:35.878 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:35.878 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:35.878 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:35.878 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:35.878 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:35.879 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:07:35.879 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:35.879 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:35.879 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:35.879 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:35.880 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:07:35.880 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:35.880 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:35.880 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:35.880 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:07:35.881 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:07:35.881 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:35.881 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:07:35.881 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:07:35.881 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:07:35.882 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:07:35.882 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/62927,v1:127.0.0.1:6803/62927]", 2021-05-12T12:07:35.882 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:07:35.882 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/62927,v1:127.0.0.1:6809/62927]", 2021-05-12T12:07:35.882 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/62927,v1:127.0.0.1:6807/62927]", 2021-05-12T12:07:35.883 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:07:35.883 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:07:35.883 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:07:35.883 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:07:35.884 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:35.884 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:07:35.884 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:35.884 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:35.885 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:07:35.885 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:07:35.885 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:35.885 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:07:35.885 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:07:35.886 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.886 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.886 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:07:35.886 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:07:35.887 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/57475,v1:127.0.0.1:6813/57475]", 2021-05-12T12:07:35.887 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:07:35.887 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:07:35.887 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:35.887 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:35.888 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:35.888 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:35.888 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:35.888 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:35.889 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:35.889 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:07:35.889 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:35.889 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:35.889 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:35.890 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:35.890 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:07:35.890 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:35.890 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:35.890 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:35.891 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:07:35.891 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:07:35.891 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:35.891 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:07:35.892 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:07:35.892 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:07:35.892 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:07:35.892 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/57475,v1:127.0.0.1:6811/57475]", 2021-05-12T12:07:35.892 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:07:35.893 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/57475,v1:127.0.0.1:6817/57475]", 2021-05-12T12:07:35.893 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/57475,v1:127.0.0.1:6815/57475]", 2021-05-12T12:07:35.893 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:07:35.893 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:07:35.893 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:07:35.894 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:07:35.894 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:35.894 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:07:35.894 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:35.895 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:35.895 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:07:35.895 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:07:35.895 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:35.895 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:07:35.896 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:07:35.896 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.896 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.896 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:07:35.896 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:07:35.897 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/63518,v1:127.0.0.1:6813/63518]", 2021-05-12T12:07:35.897 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:07:35.897 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:07:35.897 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:07:35.898 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:07:35.898 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:07:35.898 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:07:35.898 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:07:35.898 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:07:35.899 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:07:35.899 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:07:35.899 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:07:35.899 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:07:35.900 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:07:35.900 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:07:35.900 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:07:35.900 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:07:35.901 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:07:35.901 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:07:35.901 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:07:35.901 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:07:35.902 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:07:35.902 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:07:35.902 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:07:35.902 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:07:35.902 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:07:35.903 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/63518,v1:127.0.0.1:6811/63518]", 2021-05-12T12:07:35.903 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:07:35.903 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/63518,v1:127.0.0.1:6817/63518]", 2021-05-12T12:07:35.903 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/63518,v1:127.0.0.1:6815/63518]", 2021-05-12T12:07:35.903 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:07:35.904 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:07:35.904 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:07:35.904 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:07:35.904 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:07:35.905 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:07:35.905 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:07:35.905 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:07:35.905 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:07:35.906 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:07:35.906 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:07:35.906 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:07:35.906 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:07:35.907 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.907 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.907 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:07:35.907 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:07:35.907 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:07:35.908 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:07:35.908 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.908 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:07:35.908 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.908 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:07:35.909 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 41 2021-05-12T12:07:35.909 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.909 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.909 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:07:35.910 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 10 2021-05-12T12:07:35.910 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.910 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.910 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:07:35.910 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 44 2021-05-12T12:07:35.911 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.911 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.911 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.911 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:07:35.911 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 48, 2021-05-12T12:07:35.912 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:07:35.912 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:07:35.912 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.912 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:07:35.912 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:07:35.913 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:07:35.913 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.913 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.913 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:07:35.914 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:07:35.914 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:07:35.914 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.914 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.914 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:07:35.915 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:07:35.915 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:07:35.915 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.915 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.916 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:07:35.916 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.916 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:07:35.916 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:07:35.916 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.917 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.917 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:07:35.917 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:07:35.917 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.917 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.918 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:07:35.918 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:07:35.918 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.918 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.918 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:07:35.919 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:07:35.919 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.919 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.919 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:07:35.919 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:07:35.920 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.920 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.920 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:07:35.920 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:07:35.920 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.921 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.921 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:07:35.921 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:07:35.921 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.922 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.922 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:07:35.922 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:07:35.922 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.922 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.923 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:07:35.923 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:07:35.923 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.923 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.924 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:07:35.924 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:07:35.924 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.924 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.924 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:07:35.925 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:07:35.925 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.925 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.925 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:07:35.925 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:07:35.926 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.926 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.926 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:07:35.926 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.926 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:07:35.927 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:07:35.927 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:07:35.927 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:07:35.927 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.928 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:07:35.928 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:07:35.928 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:07:35.928 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.928 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:07:35.929 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.929 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:07:35.929 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.929 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.929 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.930 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.930 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:07:35.930 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:07:35.930 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:07:35.931 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:07:35.931 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.931 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:07:35.931 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:07:35.931 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:07:35.932 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.932 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:07:35.932 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.932 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:07:35.933 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.933 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.933 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.933 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.934 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:07:35.934 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:07:35.934 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:07:35.934 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:07:35.934 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.935 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:07:35.935 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:07:35.935 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:07:35.935 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.936 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:07:35.936 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.936 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:07:35.936 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.936 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.937 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:07:35.937 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.937 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:07:35.937 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.937 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.938 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:07:35.938 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.938 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:07:35.938 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.938 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.939 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.939 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.939 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:07:35.939 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:07:35.939 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:07:35.940 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.941 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:07:35.941 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.941 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:07:35.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.942 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.942 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:07:35.942 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.942 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:07:35.942 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.943 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.943 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:07:35.943 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:07:35.943 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:07:35.943 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.944 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.944 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.944 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.944 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:07:35.944 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.945 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:07:35.945 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:07:35.945 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:07:35.945 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:07:35.945 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:07:35.946 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:07:35.946 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:07:35.946 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.946 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:07:35.946 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:07:35.947 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:07:35.947 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.947 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.947 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:07:35.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:07:35.948 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:07:35.948 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.948 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.948 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:07:35.948 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.949 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:07:35.949 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.949 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:35.949 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:07:35.949 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:07:35.950 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:07:35.950 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:07:35.950 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:07:35.950 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:07:35.950 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:07:35.951 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:07:35.951 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:07:35.951 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:07:35.951 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:07:35.951 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:07:35.952 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:07:35.952 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:07:35.952 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:07:35.952 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 0, 2021-05-12T12:07:35.952 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:07:35.953 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:07:35.953 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:07:35.953 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:07:35.953 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:07:35.953 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.954 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:07:35.954 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.954 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:07:35.954 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:07:35.955 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:07:35.955 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:07:35.955 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:07:35.955 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:07:35.955 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:07:35.956 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:07:35.956 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:07:35.956 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:07:35.956 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:07:35.956 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:07:35.957 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:07:35.957 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:07:35.957 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:07:35.957 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:07:35.957 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.958 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.958 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:07:35.958 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:07:35.958 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:07:35.959 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.959 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:07:35.959 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:07:35.959 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.960 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:07:35.960 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:07:35.960 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:07:35.960 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:07:35.961 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:07:35.961 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:07:35.961 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 128, 2021-05-12T12:07:35.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 128, 2021-05-12T12:07:35.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:07:35.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:07:35.962 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:07:35.963 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:07:35.963 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:07:35.963 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:07:35.964 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:07:35.964 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:07:35.964 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:35.964 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:07:35.965 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:35.965 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:35.965 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:35.965 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:35.966 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:07:35.966 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:07:35.966 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:07:35.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:07:35.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:07:35.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:35.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:35.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:35.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:35.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:35.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:35.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:07:35.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:35.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:35.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:07:35.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:07:35.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:07:35.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:07:35.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:07:35.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:35.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:35.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:35.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:35.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:35.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:35.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:35.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:35.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:35.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:07:35.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:07:35.972 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.973 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:07:35.973 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:07:35.973 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:07:35.973 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:07:35.974 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:07:35.974 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:07:35.974 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:07:35.974 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:35.975 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:07:35.975 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:07:35.975 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:07:35.975 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.975 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 2048, 2021-05-12T12:07:35.976 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 2048, 2021-05-12T12:07:35.976 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 256, 2021-05-12T12:07:35.976 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 256, 2021-05-12T12:07:35.976 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:07:35.976 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.977 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:07:35.977 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:07:35.977 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:07:35.977 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 257, 2021-05-12T12:07:35.978 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:07:35.978 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:35.978 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:35.978 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:07:35.978 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 22940, 2021-05-12T12:07:35.979 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2236, 2021-05-12T12:07:35.979 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:07:35.979 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 20669, 2021-05-12T12:07:35.979 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314549860, 2021-05-12T12:07:35.980 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:07:35.980 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:07:35.980 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322099056640, 2021-05-12T12:07:35.980 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:07:35.980 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2289664, 2021-05-12T12:07:35.981 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 490851, 2021-05-12T12:07:35.981 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:07:35.981 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:35.981 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:07:35.982 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:07:35.982 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 21165968 2021-05-12T12:07:35.982 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.982 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:07:35.983 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:35.983 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:35.983 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:35.983 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:07:35.984 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:07:35.984 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:07:35.984 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.984 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:07:35.984 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:07:35.985 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:07:35.985 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:07:35.985 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:07:35.985 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.985 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:07:35.986 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:07:35.986 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.986 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:07:35.986 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 257, 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:07:35.987 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:07:35.988 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:07:35.988 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 22940, 2021-05-12T12:07:35.988 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2236, 2021-05-12T12:07:35.988 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:07:35.988 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 20669, 2021-05-12T12:07:35.989 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314549860, 2021-05-12T12:07:35.989 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:07:35.989 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:07:35.989 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322099056640, 2021-05-12T12:07:35.989 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:07:35.990 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2289664, 2021-05-12T12:07:35.990 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 490851, 2021-05-12T12:07:35.990 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:07:35.990 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:35.990 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:07:35.991 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:07:35.991 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 21165968 2021-05-12T12:07:35.991 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.991 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:07:35.992 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:07:35.992 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:07:35.992 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:07:35.992 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:07:35.993 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:07:35.993 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:07:35.993 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.993 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:07:35.993 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:07:35.994 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:07:35.994 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:07:35.994 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:07:35.994 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.994 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:07:35.995 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:07:35.995 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:35.995 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:35.995 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:07:35.996 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:35.996 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:07:35.996 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 128, 2021-05-12T12:07:35.996 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:07:35.996 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48300, 2021-05-12T12:07:35.997 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:07:35.997 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:07:35.997 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:07:35.997 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:07:35.997 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:07:35.998 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:07:35.998 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:07:35.998 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:07:35.998 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:07:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:07:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:07:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:07:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 100, 2021-05-12T12:07:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 100, 2021-05-12T12:07:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:07:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:07:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:07:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:07:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:07:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:07:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:07:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:07:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:07:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:07:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:07:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:07:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:07:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:07:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:07:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:07:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:07:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:07:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:07:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:07:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:07:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:07:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:07:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:07:36.006 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:07:36.006 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1220608, 2021-05-12T12:07:36.006 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 143228, 2021-05-12T12:07:36.006 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:07:36.007 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:07:36.007 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:07:36.007 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:07:36.007 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:07:36.008 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.008 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 2048, 2021-05-12T12:07:36.008 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 2048, 2021-05-12T12:07:36.008 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 256, 2021-05-12T12:07:36.009 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 256, 2021-05-12T12:07:36.009 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:07:36.009 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.009 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:36.009 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:07:36.010 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.010 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:07:36.010 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 176093659146 2021-05-12T12:07:36.010 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.010 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.011 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:07:36.011 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 42949672976 2021-05-12T12:07:36.011 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.011 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.012 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:07:36.012 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 184683593738 2021-05-12T12:07:36.012 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.012 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:36.012 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:07:36.013 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.013 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized", 2021-05-12T12:07:36.013 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 28 2021-05-12T12:07:36.013 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.014 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.014 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:07:36.014 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 100 2021-05-12T12:07:36.014 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.015 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:36.015 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:07:36.015 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.015 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:07:36.015 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:07:36.016 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 128, 2021-05-12T12:07:36.016 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:36.016 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.016 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.016 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:07:36.017 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:07:36.017 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:07:36.017 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:36.017 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.017 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.018 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:07:36.018 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 128, 2021-05-12T12:07:36.018 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 128, 2021-05-12T12:07:36.018 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:07:36.018 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.019 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:36.019 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:07:36.019 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:07:36.019 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:07:36.019 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:07:36.020 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.020 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:07:36.020 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:07:36.020 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 4, 2021-05-12T12:07:36.020 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:06:59.563453+0000", 2021-05-12T12:07:36.021 INFO:tasks.workunit.client.0.smithi090.stdout: "services": { 2021-05-12T12:07:36.021 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": { 2021-05-12T12:07:36.021 INFO:tasks.workunit.client.0.smithi090.stdout: "daemons": { 2021-05-12T12:07:36.021 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": "", 2021-05-12T12:07:36.021 INFO:tasks.workunit.client.0.smithi090.stdout: "0": { 2021-05-12T12:07:36.022 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:07:36.022 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:07:36.022 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:07:36.022 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: "2": { 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: "start_epoch": 0, 2021-05-12T12:07:36.023 INFO:tasks.workunit.client.0.smithi090.stdout: "start_stamp": "0.000000", 2021-05-12T12:07:36.024 INFO:tasks.workunit.client.0.smithi090.stdout: "gid": 0, 2021-05-12T12:07:36.024 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T12:07:36.024 INFO:tasks.workunit.client.0.smithi090.stdout: "metadata": {}, 2021-05-12T12:07:36.024 INFO:tasks.workunit.client.0.smithi090.stdout: "task_status": {} 2021-05-12T12:07:36.024 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.025 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.025 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.025 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.025 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:07:36.026 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:07:36.026 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:07:36.026 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 375, 2021-05-12T12:07:36.026 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:07:36.026 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:07:36.027 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:07:36.027 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:07:36.028 INFO:tasks.workunit.client.0.smithi090.stderr:report 600740237 2021-05-12T12:07:36.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:07:36.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2021-05-12T12:07:36.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2021-05-12T12:07:36.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2021-05-12T12:07:36.029 INFO:tasks.workunit.client.0.smithi090.stdout:writing non-divergent object existing_4 2021-05-12T12:07:36.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:07:36.095 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:07:36.095 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.096 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.096 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.096 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.096 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.097 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.097 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.097 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.097 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.097 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.098 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.098 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.098 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.098 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.099 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.099 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.099 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.099 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.100 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.100 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.100 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.100 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.101 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.101 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.101 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.101 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.101 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.102 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.102 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.102 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.102 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.103 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.103 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.103 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.103 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.104 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.104 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.104 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.104 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.105 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.105 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.105 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.105 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.106 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.106 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.106 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.106 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.107 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.107 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.107 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.107 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.108 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.108 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.108 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.108 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.109 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.109 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.109 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.109 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.109 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.110 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.110 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.110 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.110 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.110 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.111 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.111 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.111 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.111 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.112 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.112 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.112 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.112 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.113 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.113 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.113 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.113 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.113 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.114 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.114 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.114 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.114 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.114 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.115 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.115 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.115 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.115 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.116 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.116 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.116 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.116 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.116 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.117 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.117 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.117 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.117 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.118 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.118 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.118 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.118 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.118 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.119 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.119 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.119 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.119 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.120 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.121 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.121 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.121 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.121 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.122 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.122 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.122 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.122 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.122 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.123 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.123 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.123 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.123 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.124 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.124 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.124 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.124 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.124 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:07:36.125 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:07:36.125 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:07:36.126 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:07:36.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:07:36.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:07:36.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2021-05-12T12:07:36.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:07:36.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:07:36.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:07:36.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:07:36.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:07:36.242 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:07:36.243 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:07:36.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:07:36.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:07:36.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:07:36.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:07:36.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:07:36.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:07:36.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:07:36.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:07:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:07:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:07:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:07:36.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:07:36.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:36.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:07:36.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659149 2021-05-12T12:07:36.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659149 2021-05-12T12:07:36.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659149' 2021-05-12T12:07:36.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:36.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:07:36.890 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:07:36.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:07:36.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:07:36.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:07:36.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:07:36.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:07:37.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593740 2021-05-12T12:07:37.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593740 2021-05-12T12:07:37.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-176093659149 2-184683593740' 2021-05-12T12:07:37.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:37.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-176093659149 2021-05-12T12:07:37.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:37.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:07:37.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-176093659149 2021-05-12T12:07:37.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:37.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659149 2021-05-12T12:07:37.028 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 176093659149 2021-05-12T12:07:37.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 176093659149' 2021-05-12T12:07:37.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:37.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659147 -lt 176093659149 2021-05-12T12:07:37.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:07:38.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:07:38.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:07:38.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659149 -lt 176093659149 2021-05-12T12:07:38.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:07:38.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-184683593740 2021-05-12T12:07:38.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:07:38.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:07:38.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-184683593740 2021-05-12T12:07:38.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:07:38.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593740 2021-05-12T12:07:38.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 184683593740' 2021-05-12T12:07:38.787 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 184683593740 2021-05-12T12:07:38.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:07:39.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593740 -lt 184683593740 2021-05-12T12:07:39.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:07:39.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:39.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:39.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 128 == 0 2021-05-12T12:07:39.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:39.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:40.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:40.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:40.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:40.040 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:40.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:40.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:07:40.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:07:40.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:07:40.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:40.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:07:40.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:40.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:40.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:40.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:40.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:40.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:40.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:40.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:41.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:41.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:41.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:41.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:41.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:41.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:41.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:41.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:41.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:41.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:41.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:41.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:41.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:41.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:41.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:41.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:07:41.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:41.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:07:42.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:42.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:42.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:42.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:42.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:42.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:42.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:42.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:42.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:42.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:42.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:42.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:43.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:43.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:43.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:43.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:43.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:43.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:43.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:43.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:43.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:43.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:43.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:43.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:07:43.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:43.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:07:43.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:43.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:43.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:43.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:43.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:43.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:43.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:43.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:44.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:44.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:44.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:44.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:44.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:44.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:44.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:44.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:44.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:44.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:44.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:44.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:44.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:45.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:45.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:45.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:07:45.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:45.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:07:46.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:46.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:46.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:46.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:46.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:46.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:46.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:46.020 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:46.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:46.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:46.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:46.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:46.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:46.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:46.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:46.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:46.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:46.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:46.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:46.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:46.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:47.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:47.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:47.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:07:47.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:47.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:07:48.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:48.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:48.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:48.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:48.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:48.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:48.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:48.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:49.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:49.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:49.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:49.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:49.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:49.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:49.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:49.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:49.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:49.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:49.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:49.807 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:49.807 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:50.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:50.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:50.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:07:50.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:50.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:07:53.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:07:53.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:07:53.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:07:53.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:07:53.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:07:53.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:07:53.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:07:53.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:07:53.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:07:53.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:07:53.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:07:53.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:07:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:07:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:07:54.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:07:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:07:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:07:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:07:54.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:07:54.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:07:54.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:07:54.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:07:54.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:07:54.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:07:54.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:07:54.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:08:01.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:08:01.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:08:01.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:08:01.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:08:01.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:08:01.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:08:01.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:08:01.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:08:01.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:08:01.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:08:01.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:08:01.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:08:02.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:08:02.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:08:02.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:08:02.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:08:02.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:08:02.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:08:02.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:08:02.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:08:02.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:08:02.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:08:02.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:08:02.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:08:02.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:08:02.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:08:09.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:08:09.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:08:09.865 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:08:09.865 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:08:09.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:08:09.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:08:09.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:08:09.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:08:10.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:08:10.248 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:08:10.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:08:10.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:08:10.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 128 2021-05-12T12:08:10.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:08:10.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:08:10.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:08:10.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:08:10.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:08:10.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:08:10.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:08:10.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:08:11.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:08:11.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:08:11.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:08:11.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:08:11.672 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:08:11.672 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "ef34446d-c6a9-4dba-bad5-4b4f94d7bb47", 2021-05-12T12:08:11.672 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:08:11.672 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:08:11.673 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:08:11.668177+0000", 2021-05-12T12:08:11.673 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:08:11.673 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:08:11.673 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:08:11.673 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:08:11.674 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:08:11.674 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:08:11.674 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:08:11.674 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:08:11.674 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:08:11.675 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.675 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:08:11.675 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.675 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:08:11.676 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:08:11.677 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:08:11.677 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:08:11.677 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:08:11.677 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.677 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:08:11.678 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:08:11.678 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.678 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:08:11.678 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:08:11.679 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:08:11.679 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:08:11.679 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:08:11.679 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.679 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:08:11.680 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.680 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:08:11.680 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.680 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.680 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:08:11.681 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.681 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:08:11.681 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:08:11.681 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:08:11.681 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 100 pgs degraded, 128 pgs undersized", 2021-05-12T12:08:11.682 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 228 2021-05-12T12:08:11.682 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.682 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:08:11.682 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.682 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.23 is active+undersized+degraded, acting [0,2]" 2021-05-12T12:08:11.683 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.683 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.683 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.4e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.683 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.684 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.684 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.4f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.684 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.684 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.684 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.50 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.685 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.685 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.685 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.51 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.685 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.685 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.686 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.52 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.686 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.686 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.686 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.53 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.687 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.687 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.687 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.54 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.687 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.688 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.688 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.55 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.688 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.688 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.689 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.56 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.689 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.689 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.689 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.57 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.689 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.690 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.690 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.58 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.690 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.690 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.690 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.59 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.691 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.691 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.691 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.691 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.692 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.692 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.692 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.692 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.692 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.693 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.693 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.693 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.693 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.693 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.694 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.694 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.694 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.694 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.5f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.694 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.695 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.695 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.60 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.695 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.695 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.695 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.61 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.696 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.696 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.696 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.62 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.696 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.697 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.697 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.63 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.697 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.697 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.697 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.64 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-05-12T12:08:11.698 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.698 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.698 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.65 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.698 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.698 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.699 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.66 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.699 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.699 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.699 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.67 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.700 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.700 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.700 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.68 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.700 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.701 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.701 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.69 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.701 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.701 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.701 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6a is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.702 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.702 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.702 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6b is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.703 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.703 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.703 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6c is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.703 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.703 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.704 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6d is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.704 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.704 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.704 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6e is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.705 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.705 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.705 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.6f is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.705 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.706 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.706 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.70 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.706 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.706 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.707 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.71 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.707 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.707 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.707 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.72 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.708 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.708 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.708 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.73 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.709 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.709 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.709 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.74 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.710 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.710 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.710 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.75 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.710 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.711 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.711 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.76 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.711 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.712 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.712 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.77 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.712 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.712 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.713 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.78 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.713 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.713 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.713 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.79 is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.713 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.714 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.714 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7a is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.714 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.714 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.715 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7b is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.715 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.715 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.715 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7c is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.716 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.716 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.716 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7d is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.716 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.717 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.717 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7e is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.717 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.717 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.717 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.7f is stuck undersized for 69s, current state active+undersized, last acting [0,2]" 2021-05-12T12:08:11.718 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.718 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.718 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:08:11.718 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.719 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:08:11.719 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:08:11.719 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:08:11.719 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:08:11.719 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:08:11.720 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.720 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:08:11.720 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.720 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:08:11.721 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.721 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.721 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:08:11.721 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.721 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.722 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:08:11.722 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.722 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.722 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:08:11.722 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.723 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:08:11.723 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:08:11.723 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:08:11.723 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:08:11.724 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:08:11.724 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:05:37.733410Z", 2021-05-12T12:08:11.724 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:05:37.733410Z", 2021-05-12T12:08:11.724 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:08:11.724 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:08:11.725 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:08:11.725 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:08:11.725 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:08:11.725 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:08:11.725 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:08:11.726 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:08:11.726 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:08:11.726 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:08:11.726 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:08:11.727 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:08:11.727 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:08:11.727 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:08:11.727 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:08:11.728 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:08:11.728 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.728 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:08:11.728 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.728 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:08:11.729 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.729 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:08:11.729 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:08:11.729 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:08:11.729 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.730 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.730 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.730 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:08:11.730 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:08:11.731 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.731 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.731 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.731 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:08:11.732 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:08:11.732 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:08:11.732 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:08:11.732 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:08:11.732 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.733 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.733 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.733 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:08:11.733 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:08:11.734 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.734 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:08:11.734 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 48, 2021-05-12T12:08:11.734 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:08:11.734 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:05:38.600554+0000", 2021-05-12T12:08:11.735 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:07:00.175434+0000", 2021-05-12T12:08:11.735 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:06:58.573450+0000", 2021-05-12T12:08:11.735 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:06:59.655908+0000", 2021-05-12T12:08:11.735 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:08:11.736 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:08:11.736 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:08:11.736 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:08:11.736 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:08:11.736 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:08:11.737 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:08:11.737 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:08:11.737 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:08:11.737 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:08:11.737 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.738 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 9, 2021-05-12T12:08:11.738 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:08:11.738 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:08:11.738 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:08:11.739 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:08:11.739 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:08:11.739 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:08:11.739 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:08:11.739 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:08:11.740 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:08:11.740 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:08:11.740 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.740 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:08:11.740 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:08:11.741 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:06:09.184020+0000", 2021-05-12T12:08:11.741 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 1, 2021-05-12T12:08:11.741 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool", 2021-05-12T12:08:11.741 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:08:11.742 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:08:11.742 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:08:11.742 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 0, 2021-05-12T12:08:11.742 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:08:11.743 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:08:11.743 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:08:11.743 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:08:11.743 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:08:11.744 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "on", 2021-05-12T12:08:11.744 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 128, 2021-05-12T12:08:11.744 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:08:11.744 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 128, 2021-05-12T12:08:11.745 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 128, 2021-05-12T12:08:11.745 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 128, 2021-05-12T12:08:11.745 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:08:11.745 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:08:11.745 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:08:11.746 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:08:11.746 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:08:11.746 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:08:11.746 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:08:11.747 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.747 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "33", 2021-05-12T12:08:11.747 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:08:11.747 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:08:11.748 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "33", 2021-05-12T12:08:11.748 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:08:11.748 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "selfmanaged", 2021-05-12T12:08:11.748 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 0, 2021-05-12T12:08:11.748 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 0, 2021-05-12T12:08:11.749 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [], 2021-05-12T12:08:11.749 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:08:11.749 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:08:11.749 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:08:11.750 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:08:11.750 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:08:11.750 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:08:11.750 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:08:11.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:08:11.751 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:08:11.751 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:08:11.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:08:11.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:08:11.752 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:08:11.752 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:08:11.752 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:08:11.752 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "", 2021-05-12T12:08:11.753 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:08:11.753 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:08:11.753 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.753 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:08:11.753 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:08:11.754 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:08:11.754 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:08:11.754 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:08:11.755 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:08:11.755 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:08:11.755 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:08:11.755 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 0, 2021-05-12T12:08:11.755 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:08:11.756 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:08:11.756 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:08:11.756 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:08:11.756 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.757 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.757 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:08:11.757 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.757 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:08:11.757 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "caff8b80-0f3f-4420-bed0-4c96fe2cdd1e", 2021-05-12T12:08:11.758 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:08:11.758 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:08:11.758 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:08:11.758 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:08:11.758 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:08:11.759 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 31, 2021-05-12T12:08:11.759 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 41, 2021-05-12T12:08:11.759 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 47, 2021-05-12T12:08:11.759 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 35, 2021-05-12T12:08:11.760 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:08:11.760 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:08:11.760 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.760 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.760 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.761 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:08:11.761 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.761 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.761 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.761 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.762 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:08:11.762 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.762 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.762 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.763 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.763 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:08:11.763 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.763 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.764 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.764 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:08:11.764 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.764 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.764 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.765 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.765 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:08:11.765 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.765 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.765 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.766 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.766 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:08:11.766 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.766 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.767 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.767 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:08:11.767 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.767 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.767 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.768 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.768 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:08:11.768 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.768 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.769 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.769 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.769 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:08:11.769 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.770 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.770 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.770 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:08:11.770 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.771 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.771 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.771 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.771 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:08:11.771 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 62927 2021-05-12T12:08:11.772 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.772 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.772 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.772 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/62927", 2021-05-12T12:08:11.773 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/62927", 2021-05-12T12:08:11.773 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/62927", 2021-05-12T12:08:11.773 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/62927", 2021-05-12T12:08:11.773 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:08:11.773 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:08:11.774 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:08:11.774 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.774 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.774 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.775 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:08:11.775 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "7f44dd8d-eca8-43af-a678-32c354a5a07d", 2021-05-12T12:08:11.775 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:08:11.775 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:08:11.775 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:08:11.776 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:08:11.776 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:08:11.776 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:08:11.776 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 10, 2021-05-12T12:08:11.776 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 33, 2021-05-12T12:08:11.777 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 36, 2021-05-12T12:08:11.777 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:08:11.777 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:08:11.777 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.777 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.778 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.778 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:08:11.778 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.778 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.779 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.779 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.779 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:08:11.779 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.779 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.780 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.780 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.780 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:08:11.780 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.781 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.781 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.781 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:08:11.781 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.781 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.782 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.782 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.782 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:08:11.782 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.782 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.783 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.783 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.783 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:08:11.783 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.784 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.784 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.784 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:08:11.784 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.785 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.785 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.785 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.785 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:08:11.785 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.786 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.786 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.786 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.786 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:08:11.787 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.787 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.787 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.787 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:08:11.787 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.788 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.788 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.788 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.788 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:08:11.789 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 57475 2021-05-12T12:08:11.789 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.789 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.789 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.790 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/57475", 2021-05-12T12:08:11.790 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/57475", 2021-05-12T12:08:11.790 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/57475", 2021-05-12T12:08:11.790 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/57475", 2021-05-12T12:08:11.791 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:08:11.791 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:08:11.791 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.791 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.791 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.792 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:08:11.792 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "9fb1ffaa-ed93-42e3-9c56-62b59ef90144", 2021-05-12T12:08:11.792 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:08:11.792 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:08:11.792 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:08:11.793 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:08:11.793 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 15, 2021-05-12T12:08:11.793 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 31, 2021-05-12T12:08:11.793 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 43, 2021-05-12T12:08:11.793 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:08:11.794 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 37, 2021-05-12T12:08:11.794 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:08:11.794 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:08:11.794 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.794 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.795 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.795 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:08:11.795 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.795 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.796 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.796 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.796 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:08:11.796 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.796 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.797 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.797 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.797 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:08:11.797 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.797 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.798 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.798 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:08:11.798 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.798 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.799 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.799 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.799 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:08:11.799 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.799 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.800 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.800 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.800 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:08:11.800 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.800 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.801 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.801 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:08:11.801 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.801 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.802 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.802 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.802 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:08:11.802 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.802 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.803 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.803 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.803 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:08:11.803 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:08:11.803 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.804 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:08:11.804 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:08:11.804 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.804 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.805 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.805 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:08:11.805 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:08:11.805 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 63518 2021-05-12T12:08:11.805 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.806 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.806 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.806 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/63518", 2021-05-12T12:08:11.806 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/63518", 2021-05-12T12:08:11.807 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/63518", 2021-05-12T12:08:11.807 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/63518", 2021-05-12T12:08:11.807 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:08:11.807 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:08:11.807 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:08:11.808 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.808 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.808 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.808 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:08:11.808 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.809 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:08:11.809 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:47.561716+0000", 2021-05-12T12:08:11.809 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:08:11.809 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:08:11.810 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:08:11.810 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:08:11.810 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:48.458088+0000", 2021-05-12T12:08:11.810 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:08:11.810 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.811 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.811 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:08:11.811 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:48.077947+0000", 2021-05-12T12:08:11.811 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:08:11.812 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:08:11.812 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:08:11.812 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:08:11.812 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:05:57.533501+0000", 2021-05-12T12:08:11.812 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:08:11.813 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.813 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.813 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:08:11.813 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:06:48.594152+0000", 2021-05-12T12:08:11.813 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:08:11.814 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:08:11.814 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:08:11.814 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:08:11.814 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:06:06.850113+0000", 2021-05-12T12:08:11.814 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:08:11.815 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.815 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.815 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:08:11.815 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:08:11.816 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:08:11.816 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:08:11.816 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:08:11.816 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:08:11.816 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:08:11.817 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:08:11.817 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:08:11.817 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:08:11.817 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:08:11.817 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.818 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.818 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:08:11.818 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:08:11.818 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:08:11.819 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:08:11.819 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:08:11.819 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:08:11.819 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:08:11.819 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:08:11.820 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:08:11.820 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:08:11.820 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:08:11.820 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.820 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/62927,v1:127.0.0.1:6805/62927]", 2021-05-12T12:08:11.821 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:08:11.822 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:08:11.822 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:08:11.822 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:08:11.822 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:08:11.822 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:08:11.823 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:08:11.823 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:08:11.823 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:08:11.823 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:08:11.824 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:08:11.824 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:08:11.824 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:08:11.824 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:08:11.825 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:08:11.826 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:08:11.827 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:08:11.827 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/62927,v1:127.0.0.1:6803/62927]", 2021-05-12T12:08:11.827 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:08:11.827 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/62927,v1:127.0.0.1:6809/62927]", 2021-05-12T12:08:11.827 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/62927,v1:127.0.0.1:6807/62927]", 2021-05-12T12:08:11.828 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:08:11.828 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:08:11.828 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:08:11.828 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:08:11.828 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:08:11.829 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:08:11.829 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:08:11.829 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:08:11.829 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:08:11.830 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:08:11.830 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:08:11.830 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:08:11.830 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:08:11.830 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.831 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.831 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:08:11.831 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:08:11.832 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/57475,v1:127.0.0.1:6813/57475]", 2021-05-12T12:08:11.832 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:08:11.832 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:08:11.832 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:08:11.832 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:08:11.833 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:08:11.833 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:08:11.833 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:08:11.833 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:08:11.833 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:08:11.834 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:08:11.834 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:08:11.834 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:08:11.834 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:08:11.835 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:08:11.835 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:08:11.835 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:08:11.835 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:08:11.836 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:08:11.836 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:08:11.836 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:08:11.836 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:08:11.836 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:08:11.837 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:08:11.837 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:08:11.837 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:08:11.837 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/57475,v1:127.0.0.1:6811/57475]", 2021-05-12T12:08:11.837 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:08:11.838 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/57475,v1:127.0.0.1:6817/57475]", 2021-05-12T12:08:11.838 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/57475,v1:127.0.0.1:6815/57475]", 2021-05-12T12:08:11.838 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:08:11.838 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:08:11.839 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:08:11.839 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:08:11.839 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:08:11.839 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:08:11.839 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:08:11.840 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:08:11.840 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:08:11.840 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:08:11.840 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:08:11.841 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:08:11.841 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:08:11.841 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.841 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.841 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:08:11.842 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:08:11.842 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/63518,v1:127.0.0.1:6813/63518]", 2021-05-12T12:08:11.842 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:08:11.842 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:08:11.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:08:11.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:08:11.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:08:11.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:08:11.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:08:11.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:08:11.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:08:11.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:08:11.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:08:11.845 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:08:11.845 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:08:11.845 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:08:11.845 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:08:11.846 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:08:11.846 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:08:11.846 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:08:11.846 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:08:11.846 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:08:11.847 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:08:11.847 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:08:11.847 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:08:11.847 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:08:11.848 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:08:11.848 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/63518,v1:127.0.0.1:6811/63518]", 2021-05-12T12:08:11.848 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:08:11.848 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/63518,v1:127.0.0.1:6817/63518]", 2021-05-12T12:08:11.848 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/63518,v1:127.0.0.1:6815/63518]", 2021-05-12T12:08:11.849 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:08:11.849 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:08:11.849 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:08:11.849 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:08:11.849 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:08:11.850 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:08:11.850 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:08:11.850 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:08:11.850 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:08:11.850 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:08:11.851 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:08:11.851 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:08:11.851 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:08:11.851 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.851 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.852 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:08:11.852 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:08:11.852 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:08:11.852 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:08:11.852 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.853 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:08:11.853 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.853 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:08:11.853 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 41 2021-05-12T12:08:11.854 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.854 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.854 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:08:11.854 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 10 2021-05-12T12:08:11.854 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.855 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.855 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:08:11.855 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 44 2021-05-12T12:08:11.855 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.855 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.856 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.856 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:08:11.856 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 48, 2021-05-12T12:08:11.856 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:08:11.856 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:08:11.857 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.857 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:08:11.857 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:08:11.857 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:08:11.857 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.858 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.858 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:08:11.858 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:08:11.858 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:08:11.858 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.859 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.859 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:08:11.859 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:08:11.859 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:08:11.859 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.860 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.860 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:08:11.860 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.860 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:08:11.860 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:08:11.861 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.861 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.861 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:08:11.861 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:08:11.861 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.862 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.862 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:08:11.862 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:08:11.862 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.863 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.863 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:08:11.863 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:08:11.863 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.863 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:08:11.864 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:08:11.865 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.865 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.865 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:08:11.865 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:08:11.865 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.866 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.866 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:08:11.866 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:08:11.866 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.866 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.867 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:08:11.867 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:08:11.867 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.867 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.867 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:08:11.868 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:08:11.868 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.868 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.868 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:08:11.868 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:08:11.869 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.869 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.869 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:08:11.869 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:08:11.870 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.870 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.870 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:08:11.870 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.870 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:08:11.871 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:08:11.872 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:08:11.872 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.872 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:08:11.872 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.872 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:08:11.873 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:08:11.874 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:08:11.874 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:08:11.874 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.874 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:08:11.874 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:08:11.875 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:08:11.875 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.875 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:08:11.875 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.875 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:08:11.876 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.876 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.876 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.876 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:08:11.877 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:08:11.877 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:08:11.877 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:08:11.877 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.878 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:08:11.878 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:08:11.878 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:08:11.878 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.878 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:08:11.879 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.879 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:08:11.879 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.879 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.880 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:08:11.880 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.880 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:08:11.880 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.880 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.881 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:08:11.881 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.881 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:08:11.881 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.882 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.882 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.882 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.882 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:08:11.882 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:08:11.883 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:08:11.883 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:08:11.883 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:08:11.883 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:08:11.883 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:08:11.884 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:08:11.884 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.884 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:08:11.888 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.888 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:08:11.888 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.888 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.889 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:08:11.889 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.889 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:08:11.889 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.889 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.890 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:08:11.890 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:08:11.890 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:08:11.890 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.891 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.891 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.891 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.891 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:08:11.891 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.892 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:08:11.892 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:08:11.892 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:08:11.892 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:08:11.893 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:08:11.893 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:08:11.893 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:08:11.893 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.893 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:08:11.894 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:08:11.894 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:08:11.894 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.894 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.895 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:08:11.895 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:08:11.895 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:08:11.895 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.895 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.896 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:08:11.896 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.896 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:08:11.896 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.896 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.897 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:08:11.897 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:08:11.897 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:08:11.897 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:08:11.897 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:08:11.898 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:08:11.898 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:08:11.898 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:08:11.898 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:08:11.899 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:08:11.899 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:08:11.899 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:08:11.899 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:08:11.899 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:08:11.900 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:08:11.900 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 0, 2021-05-12T12:08:11.900 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:08:11.900 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:08:11.900 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:08:11.901 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:08:11.901 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:08:11.901 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.901 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:08:11.901 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.902 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:08:11.902 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:08:11.902 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:08:11.902 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:08:11.902 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:08:11.903 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:08:11.903 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:08:11.903 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:08:11.903 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:08:11.904 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:08:11.904 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:08:11.904 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:08:11.904 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:08:11.904 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:08:11.905 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:08:11.905 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:08:11.905 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.905 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.905 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:08:11.906 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:08:11.906 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:08:11.906 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.906 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:08:11.907 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:08:11.907 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.907 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:08:11.907 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:08:11.907 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:08:11.908 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:08:11.908 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:08:11.908 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:08:11.908 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.908 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 128, 2021-05-12T12:08:11.909 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 128, 2021-05-12T12:08:11.909 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:08:11.909 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:08:11.909 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:08:11.910 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:08:11.910 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 47947, 2021-05-12T12:08:11.910 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:08:11.910 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:08:11.911 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:08:11.911 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:08:11.911 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:08:11.911 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:08:11.912 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:08:11.912 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:08:11.912 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:08:11.912 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:08:11.913 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:08:11.913 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:08:11.913 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:08:11.913 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:08:11.913 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:08:11.914 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:08:11.914 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:08:11.914 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:08:11.914 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:08:11.915 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:08:11.915 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:08:11.915 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:08:11.915 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:08:11.916 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:08:11.916 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:08:11.916 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:08:11.916 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:08:11.916 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:08:11.917 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:08:11.917 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:08:11.917 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:08:11.917 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:08:11.917 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:08:11.918 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:08:11.918 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:08:11.918 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:08:11.918 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:08:11.918 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:08:11.919 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:08:11.919 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.919 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:08:11.919 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:08:11.920 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:08:11.921 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:08:11.921 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:08:11.921 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:08:11.921 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.921 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 2049, 2021-05-12T12:08:11.922 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 2049, 2021-05-12T12:08:11.922 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 256, 2021-05-12T12:08:11.922 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 256, 2021-05-12T12:08:11.922 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:08:11.922 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:08:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:08:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:08:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 257, 2021-05-12T12:08:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:08:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:08:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:08:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:08:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 22940, 2021-05-12T12:08:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2236, 2021-05-12T12:08:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:08:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 20669, 2021-05-12T12:08:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314549860, 2021-05-12T12:08:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:08:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:08:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322099056640, 2021-05-12T12:08:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:08:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2289664, 2021-05-12T12:08:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 490145, 2021-05-12T12:08:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:08:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:08:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:08:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:08:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 21165968 2021-05-12T12:08:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:08:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:08:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:08:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:08:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:08:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:08:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:08:11.930 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.930 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:08:11.930 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:08:11.930 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:08:11.930 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:08:11.931 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:08:11.931 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.931 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:08:11.931 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:08:11.932 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.932 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:08:11.932 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:08:11.932 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:08:11.933 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:08:11.933 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 257, 2021-05-12T12:08:11.933 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:08:11.933 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:08:11.933 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:08:11.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:08:11.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 22940, 2021-05-12T12:08:11.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2236, 2021-05-12T12:08:11.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:08:11.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 20669, 2021-05-12T12:08:11.935 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314549860, 2021-05-12T12:08:11.935 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:08:11.935 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:08:11.935 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322099056640, 2021-05-12T12:08:11.936 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:08:11.936 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2289664, 2021-05-12T12:08:11.936 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 490145, 2021-05-12T12:08:11.936 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:08:11.936 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:08:11.937 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:08:11.937 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:08:11.937 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 21165968 2021-05-12T12:08:11.937 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.937 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:08:11.938 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:08:11.938 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:08:11.938 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:08:11.938 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:08:11.939 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:08:11.939 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:08:11.939 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.939 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:08:11.939 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:08:11.940 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:08:11.940 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:08:11.940 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:08:11.940 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.940 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:08:11.941 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:08:11.941 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.941 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:08:11.941 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.942 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:08:11.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 128, 2021-05-12T12:08:11.942 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:08:11.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 47947, 2021-05-12T12:08:11.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 100, 2021-05-12T12:08:11.943 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 0, 2021-05-12T12:08:11.943 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 300, 2021-05-12T12:08:11.943 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:08:11.943 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:08:11.943 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 100, 2021-05-12T12:08:11.944 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:08:11.944 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:08:11.944 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 100, 2021-05-12T12:08:11.944 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:08:11.944 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:08:11.945 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:08:11.945 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:08:11.945 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:08:11.945 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:08:11.945 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:08:11.946 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:08:11.946 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:08:11.946 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:08:11.946 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:08:11.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:08:11.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:08:11.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:08:11.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:08:11.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:08:11.948 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:08:11.948 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:08:11.948 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:08:11.948 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:08:11.949 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:08:11.949 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:08:11.949 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:08:11.949 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:08:11.949 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:08:11.950 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:08:11.950 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:08:11.950 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:08:11.950 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:08:11.950 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:08:11.951 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.951 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:08:11.951 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:08:11.951 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:08:11.951 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:08:11.952 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1220608, 2021-05-12T12:08:11.952 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 142522, 2021-05-12T12:08:11.952 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:08:11.952 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:08:11.952 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:08:11.953 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:08:11.953 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:08:11.953 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.953 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 2049, 2021-05-12T12:08:11.954 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 2049, 2021-05-12T12:08:11.954 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 256, 2021-05-12T12:08:11.954 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 256, 2021-05-12T12:08:11.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:08:11.954 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.955 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.955 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:08:11.955 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.955 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:08:11.956 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 176093659156 2021-05-12T12:08:11.956 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.956 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.956 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:08:11.956 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 42949672976 2021-05-12T12:08:11.957 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.957 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.957 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:08:11.957 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 184683593747 2021-05-12T12:08:11.957 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.958 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:08:11.958 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.958 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized", 2021-05-12T12:08:11.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 28 2021-05-12T12:08:11.959 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.959 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.959 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:08:11.959 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 100 2021-05-12T12:08:11.960 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.960 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.960 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:08:11.960 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.960 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:08:11.961 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:08:11.961 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 128, 2021-05-12T12:08:11.961 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:08:11.961 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.961 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.962 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:08:11.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:08:11.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:08:11.962 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:08:11.962 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.963 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.963 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:08:11.963 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 128, 2021-05-12T12:08:11.963 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 128, 2021-05-12T12:08:11.964 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:08:11.964 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.964 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.964 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:08:11.964 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:08:11.965 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:08:11.965 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:08:11.965 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.965 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:08:11.965 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:08:11.966 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 6, 2021-05-12T12:08:11.966 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:07:59.582688+0000", 2021-05-12T12:08:11.966 INFO:tasks.workunit.client.0.smithi090.stdout: "services": {} 2021-05-12T12:08:11.966 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:08:11.966 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:08:11.967 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:08:11.967 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 470, 2021-05-12T12:08:11.967 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:08:11.967 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:08:11.967 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:08:11.968 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:08:11.969 INFO:tasks.workunit.client.0.smithi090.stderr:report 1357037108 2021-05-12T12:08:11.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:08:11.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2021-05-12T12:08:11.969 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery 2021-05-12T12:08:11.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:12.026 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:12.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 48'101 48:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.053 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.053 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.053 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.053 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.053 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.054 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.054 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.054 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:12.054 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:12.056 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:12.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:08:12.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2021-05-12T12:08:12.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:08:12.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:08:12.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:12.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:12.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:12.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:08:12.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:08:12.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:08:12.132 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:08:12.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T12:08:12.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2021-05-12T12:08:12.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:08:12.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:08:12.143 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:12.143 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:12.143 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:12.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:08:12.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:08:12.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:08:12.237 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:08:12.245 INFO:tasks.workunit.client.0.smithi090.stdout:revive divergent 1 2021-05-12T12:08:12.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2021-05-12T12:08:12.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:12.590 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:12.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 48'101 48:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.619 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.619 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.619 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.619 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.620 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.620 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.620 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.620 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.621 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.621 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 48:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.621 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.621 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.622 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 48:123 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:12.622 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:12.622 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:12.623 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:12.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2021-05-12T12:08:13.262 INFO:tasks.workunit.client.0.smithi090.stderr:noup is set 2021-05-12T12:08:13.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-05-12T12:08:13.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:08:13.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:08:13.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:08:13.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:08:13.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:08:13.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:13.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:13.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:13.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:13.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:08:13.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:08:13.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:13.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:08:13.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:13.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:08:13.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:13.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:13.279 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:13.279 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:13.279 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:13.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:13.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:13.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:13.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:13.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:13.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:13.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:13.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:13.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:08:13.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:08:13.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:08:13.282 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:08:13.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:08:13.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:13.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:08:13.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:08:13.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:08:13.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:08:13.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:08:13.302 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:13.299+0000 7f43dc56d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:13.315 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:13.313+0000 7f43dc56d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:13.316 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:13.314+0000 7f43dc56d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:13.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:08:13.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2021-05-12T12:08:13.832 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:13.830+0000 7f43dc56d0c0 -1 Falling back to public interface 2021-05-12T12:08:16.664 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:16.661+0000 7f43dc56d0c0 -1 osd.1 34 log_to_monitors {default=true} 2021-05-12T12:08:18.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2021-05-12T12:08:18.667 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery divergent 2021-05-12T12:08:18.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:19.023 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:19.023 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.024 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.024 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.024 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.024 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.025 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.025 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.025 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.025 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.025 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.026 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.027 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.028 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.029 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.030 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.031 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.032 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.033 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.034 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.036 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.037 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.038 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.040 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.041 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.042 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 48'101 49:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.043 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.044 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.045 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.047 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.048 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.049 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.050 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.051 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 49:127 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.052 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 49:125 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:19.052 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:19.053 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:19.054 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:19.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2021-05-12T12:08:19.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:08:19.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:08:19.055 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:19.055 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:19.056 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:19.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:08:19.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:08:19.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:08:19.129 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:08:19.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2021-05-12T12:08:19.811 INFO:tasks.workunit.client.0.smithi090.stderr:noup is unset 2021-05-12T12:08:19.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2021-05-12T12:08:19.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:19.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:08:19.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:19.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:19.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:19.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:19.823 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:19.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:19.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:08:20.331 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 41 up_thru 47 down_at 35 last_clean_interval [5,31) [v2:127.0.0.1:6802/62927,v1:127.0.0.1:6803/62927] [v2:127.0.0.1:6804/62927,v1:127.0.0.1:6805/62927] exists,up caff8b80-0f3f-4420-bed0-4c96fe2cdd1e 2021-05-12T12:08:20.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:20.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:20.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:20.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2021-05-12T12:08:20.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:20.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:20.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:20.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:20.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:20.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:20.335 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:20.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:20.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:20.737 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/66208,v1:127.0.0.1:6819/66208] [v2:127.0.0.1:6820/66208,v1:127.0.0.1:6821/66208] exists,up 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:08:20.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:20.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:20.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:20.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2021-05-12T12:08:20.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:20.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:08:20.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:20.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:20.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:20.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:20.741 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:20.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:20.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:08:21.193 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 43 up_thru 0 down_at 37 last_clean_interval [15,31) [v2:127.0.0.1:6810/63518,v1:127.0.0.1:6811/63518] [v2:127.0.0.1:6812/63518,v1:127.0.0.1:6813/63518] exists,up 9fb1ffaa-ed93-42e3-9c56-62b59ef90144 2021-05-12T12:08:21.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:21.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:21.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:21.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:21.544 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:21.544 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.545 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.545 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.545 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.545 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.546 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.546 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.546 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.546 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.547 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.547 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.547 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.547 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.548 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.548 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.548 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.548 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.549 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.549 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.549 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.549 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.549 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.550 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.550 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.550 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.550 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.551 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.551 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.551 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.551 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.552 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.552 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.552 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.552 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.552 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.553 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.553 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.553 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.553 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.554 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.554 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.554 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.554 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.555 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.555 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.555 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.555 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.555 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.556 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.556 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.556 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.556 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.557 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.557 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.557 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.557 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.557 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.558 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.559 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.559 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.559 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.559 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.560 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.560 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.560 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.560 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.560 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.561 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.561 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.561 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.561 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.561 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.562 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.562 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.562 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.562 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.563 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.563 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.563 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.563 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.563 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 48'101 50:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.564 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.564 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.564 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.564 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.565 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.565 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.565 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.565 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.565 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.566 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.566 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.566 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.566 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.566 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.567 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.567 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.567 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.567 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.567 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.568 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.568 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.568 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.568 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.568 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.569 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.569 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.569 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.569 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.569 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.570 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.570 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.570 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.570 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.571 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.571 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.571 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.571 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.571 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.572 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.572 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.572 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.572 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.572 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.573 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:21.573 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:21.573 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:21.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2021-05-12T12:08:21.574 INFO:tasks.workunit.client.0.smithi090.stdout:wait for peering 2021-05-12T12:08:21.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:21.922 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:21.922 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797101+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.923 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797061+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.923 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798614+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.923 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796189+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.924 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.797002+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.924 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795050+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.924 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892363+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.924 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796350+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.925 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891836+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.925 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.795185+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.925 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796759+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.925 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796291+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.925 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798306+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.926 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716218+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.926 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796694+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.926 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716060+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.926 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.892063+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.927 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716270+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.927 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.891509+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.927 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717447+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.927 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796435+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.928 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717282+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.928 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796982+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.928 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.717039+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.929 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.796393+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.929 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.716859+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.929 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.798148+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.929 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717092+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.930 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796570+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.930 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719351+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.930 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795238+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.930 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718912+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.931 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796649+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.931 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718171+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.931 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796093+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.931 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718629+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.931 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796674+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.932 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718720+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.932 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794860+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.932 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722532+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.932 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794837+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.933 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724339+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.933 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.796051+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.933 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719871+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.933 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790878+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.934 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720777+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.934 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795272+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.934 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721586+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.934 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691070+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.935 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718172+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.935 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717900+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.935 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792512+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.935 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720112+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.936 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723086+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.936 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720909+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.936 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723546+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.936 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719113+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.937 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790432+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.937 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719816+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.937 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794371+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.937 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719680+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.937 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794446+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.938 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722980+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.938 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794559+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.938 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790908+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.938 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.795089+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.939 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790494+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.939 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724379+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.939 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721759+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.939 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793432+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.940 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793038+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.940 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792131+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.940 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723851+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.940 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722689+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.941 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+undersized 2021-05-12T12:07:00.723157+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.941 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792664+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.941 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792119+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.941 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793442+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.941 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723630+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.942 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.792984+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.942 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722770+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.942 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791682+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.942 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722295+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723414+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 1 0 0 25 0 0 17 17 active+undersized+degraded 2021-05-12T12:07:00.790939+0000 48'101 50:129 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722379+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.793046+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.943 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717817+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.944 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724071+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.944 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717641+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.944 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794782+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.944 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718088+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.944 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794248+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.945 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718571+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.945 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 1 0 0 378 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.794052+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.945 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720685+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.945 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.791297+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.945 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720219+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.946 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.723193+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.946 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719490+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.946 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721989+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.946 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.724062+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.946 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.721359+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.947 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720500+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.947 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720254+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.947 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718316+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.947 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.790487+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.947 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718230+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.948 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.722765+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.948 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717867+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.948 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719276+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.948 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718508+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.948 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719041+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.949 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718960+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.949 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.718721+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.949 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716692+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.949 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.720554+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.950 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716458+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.950 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.719735+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.950 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.716633+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.950 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717498+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.950 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.587734+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.951 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717258+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.951 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.691077+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.951 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717416+0000 31'100 50:128 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.951 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715580+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.951 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.717682+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.952 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 1 0 0 377 0 0 16 16 active+undersized+degraded 2021-05-12T12:07:00.715579+0000 31'100 50:126 [0,2] 0 [0,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:21.952 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:21.952 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:21.953 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:21.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2021-05-12T12:08:28.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2021-05-12T12:08:28.900 INFO:tasks.workunit.client.0.smithi090.stdout:killing divergent 1 2021-05-12T12:08:28.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:29.263 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:29.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.923679+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001372+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110665+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110283+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113946+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922873+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110212+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001139+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003827+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997894+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.118025+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000968+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001886+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.915712+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113652+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991039+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.109922+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.918403+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110570+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.918334+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.923462+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.914655+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 1 1 0 0 377 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:21.112914+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998697+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.993554+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116018+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001585+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991035+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003607+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114327+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.923242+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996529+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001384+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.999603+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.993328+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116409+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001218+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996410+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.004022+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925412+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.273 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922665+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.273 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997552+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.273 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000906+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.273 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000875+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113327+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922158+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000786+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997399+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.914041+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.275 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114976+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.275 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.995991+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.275 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002831+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.275 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean+wait 2021-05-12T12:08:20.924986+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.276 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.116998+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.276 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997026+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.276 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998491+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.276 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000232+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.277 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.115414+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.277 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:20.929452+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.277 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000518+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.277 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996824+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.277 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003428+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.278 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925837+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.278 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.115863+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.278 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998226+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.278 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002094+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.279 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002276+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.279 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.992990+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.279 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:21.118502+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.279 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.000123+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.279 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000026+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.280 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111021+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.280 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.991734+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.280 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.117814+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.280 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998110+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.280 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999764+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.281 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999040+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.281 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922377+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.281 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991502+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.281 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003050+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.282 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997861+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.282 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998811+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.282 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean+wait 2021-05-12T12:08:21.115627+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.282 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991135+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.283 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 1 1 0 0 25 0 0 14 14 active+recovering+degraded+wait 2021-05-12T12:08:21.112619+0000 48'101 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.283 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111543+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.283 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999961+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.283 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.921069+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.283 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.992049+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.284 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999193+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.284 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000646+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.284 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.924661+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.284 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922545+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.285 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114767+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.285 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003250+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.285 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997192+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.285 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998609+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.285 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921800+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.286 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.926696+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.286 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000485+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.286 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111013+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.286 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.997331+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.287 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.117540+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.287 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925200+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.287 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001092+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.287 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996204+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.287 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.998014+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.288 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116610+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.288 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925626+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.288 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.999395+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.288 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996615+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.289 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996325+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.289 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921290+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.289 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114540+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.289 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999854+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.289 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921680+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.290 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996784+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.290 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116210+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.290 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922570+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.290 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.919347+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.995774+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.913806+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.918356+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 0 0 0 377 0 0 11 11 active+clean+wait 2021-05-12T12:08:20.914650+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.291 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998987+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.292 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.848044+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.292 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.919394+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.292 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean+wait 2021-05-12T12:08:21.115332+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:29.292 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:29.293 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:08:29.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:29.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:29.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:29.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:29.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:29.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:29.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.29659.out 2021-05-12T12:08:29.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.29659.out 2021-05-12T12:08:29.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:08:29.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:08:29.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:08:29.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:08:29.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:08:29.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.29659.out 2021-05-12T12:08:30.219 INFO:tasks.workunit.client.0.smithi090.stderr:Exporting 1.0 info 1.0( v 31'100 (31'84,31'100] local-lis/les=53/54 n=0 ec=23/23 lis/c=53/53 les/c/f=54/54/0 sis=53 pruub=7.828119755s) 2021-05-12T12:08:30.222 INFO:tasks.workunit.client.0.smithi090.stderr:Export successful 2021-05-12T12:08:30.222 INFO:tasks.workunit.client.0.smithi090.stdout: marking collection for removal 2021-05-12T12:08:30.223 INFO:tasks.workunit.client.0.smithi090.stdout:setting '_remove' omap key 2021-05-12T12:08:30.223 INFO:tasks.workunit.client.0.smithi090.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:08:30.336 INFO:tasks.workunit.client.0.smithi090.stderr:Remove successful 2021-05-12T12:08:30.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.29659.out 2021-05-12T12:08:30.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:08:30.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:08:30.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:08:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:08:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:08:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.29659.out 2021-05-12T12:08:31.371 INFO:tasks.workunit.client.0.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e54 pg_nums {1={23=1,33=128}} deleted_pools ) 2021-05-12T12:08:31.371 INFO:tasks.workunit.client.0.smithi090.stdout:Importing pgid 1.0 2021-05-12T12:08:31.372 INFO:tasks.workunit.client.0.smithi090.stdout:write_pg epoch 54 info 1.0( v 31'100 (31'84,31'100] local-lis/les=53/54 n=0 ec=23/23 lis/c=53/53 les/c/f=54/54/0 sis=53 pruub=7.828119755s) 2021-05-12T12:08:31.430 INFO:tasks.workunit.client.0.smithi090.stdout:Import successful 2021-05-12T12:08:31.887 INFO:tasks.workunit.client.0.smithi090.stdout:reviving divergent 1 2021-05-12T12:08:31.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2021-05-12T12:08:31.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:32.242 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:32.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.923679+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.242 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001372+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110665+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110283+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113946+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.243 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922873+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.244 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110212+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.244 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001139+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.244 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003827+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.244 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997894+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.245 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.118025+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.245 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000968+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.245 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001886+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.245 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.915712+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.245 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113652+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.246 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991039+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.246 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.109922+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.246 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.918403+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.246 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.110570+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.247 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.918334+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.247 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.923462+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.247 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.914655+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.247 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 1 1 0 0 377 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:21.112914+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.247 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998697+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.248 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.993554+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.248 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116018+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.248 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001585+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.249 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991035+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.249 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003607+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.249 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114327+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.249 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.923242+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.249 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996529+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.250 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001384+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.250 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.999603+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.250 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.993328+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.250 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116409+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.251 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001218+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.251 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996410+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.251 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.004022+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.251 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925412+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.251 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922665+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.252 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997552+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.252 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000906+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.252 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000875+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.252 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.113327+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.253 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922158+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.253 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000786+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.253 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997399+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.253 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.914041+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.253 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114976+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.254 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.995991+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.254 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002831+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.254 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean+wait 2021-05-12T12:08:20.924986+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.254 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.116998+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.255 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997026+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.255 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998491+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.255 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000232+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.255 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.115414+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.256 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:20.929452+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.256 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000518+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.256 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996824+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.256 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003428+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.257 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925837+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.257 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.115863+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.257 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998226+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.257 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002094+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.257 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.002276+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.258 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.992990+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.258 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 1 1 0 0 378 0 0 16 16 active+recovering+degraded+wait 2021-05-12T12:08:21.118502+0000 31'100 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.258 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.000123+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.258 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000026+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.259 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111021+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.259 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.991734+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.259 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.117814+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.259 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998110+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.260 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999764+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.260 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999040+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.260 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922377+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.260 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991502+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.261 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003050+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.261 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997861+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.261 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998811+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.261 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean+wait 2021-05-12T12:08:21.115627+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.262 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.991135+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.262 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 1 1 0 0 25 0 0 14 14 active+recovering+degraded+wait 2021-05-12T12:08:21.112619+0000 48'101 54:133 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.262 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111543+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.262 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999961+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.262 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 0 0 0 378 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.921069+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.263 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.992049+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.263 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999193+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.263 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000646+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.924661+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922545+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114767+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.003250+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.264 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.997192+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998609+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921800+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.926696+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.265 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.000485+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.111013+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.997331+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:21.117540+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925200+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.266 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.001092+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996204+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.998014+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116610+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.925626+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.267 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.999395+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996615+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996325+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921290+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.114540+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.268 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.999854+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.921680+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.996784+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:21.116210+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.269 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.922570+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.919347+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.995774+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.270 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.913806+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 0 0 0 377 0 0 10 10 active+clean+wait 2021-05-12T12:08:20.918356+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 1 0 0 0 0 377 0 0 11 11 active+clean+wait 2021-05-12T12:08:20.914650+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.998987+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.848044+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.271 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 1 0 0 0 0 377 0 0 16 16 active+clean+wait 2021-05-12T12:08:20.919394+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.272 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean+wait 2021-05-12T12:08:21.115332+0000 31'100 54:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:32.272 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:32.272 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:32.276 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:32.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-05-12T12:08:32.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:08:32.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:08:32.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:08:32.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:08:32.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:08:32.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:32.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:32.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:32.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:32.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:08:32.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:08:32.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:32.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:08:32.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:32.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:08:32.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:32.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:32.280 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:32.280 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:32.280 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:32.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:32.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:32.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:32.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:32.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:32.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:32.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:32.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:32.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:08:32.283 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:08:32.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:08:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:08:32.286 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:32.280+0000 7fcabdac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:32.282+0000 7fcabdac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:32.287 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:32.284+0000 7fcabdac30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:32.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:08:32.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:32.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:32.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:32.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:32.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:32.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:32.627 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:32.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:32.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:32.806 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:32.804+0000 7fcabdac30c0 -1 Falling back to public interface 2021-05-12T12:08:32.986 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/66208,v1:127.0.0.1:6819/66208] [v2:127.0.0.1:6820/66208,v1:127.0.0.1:6821/66208] exists,up 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:08:32.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:32.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:32.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:32.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2021-05-12T12:08:32.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:08:32.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:08:32.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:08:32.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:08:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:08:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:08:32.989 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:08:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:08:33.377 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 36 last_clean_interval [10,34) [v2:127.0.0.1:6818/66208,v1:127.0.0.1:6819/66208] [v2:127.0.0.1:6820/66208,v1:127.0.0.1:6821/66208] exists,up 7f44dd8d-eca8-43af-a678-32c354a5a07d 2021-05-12T12:08:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:08:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:08:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:08:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2021-05-12T12:08:35.215 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:35.212+0000 7fcabdac30c0 -1 osd.1 54 log_to_monitors {default=true} 2021-05-12T12:08:53.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2021-05-12T12:08:53.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:08:53.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:08:53.381 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:53.381 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:53.381 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:53.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:08:53.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2021-05-12T12:08:53.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok dump_ops_in_flight 2021-05-12T12:08:53.482 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:08:53.482 INFO:tasks.workunit.client.0.smithi090.stdout: "ops": [], 2021-05-12T12:08:53.482 INFO:tasks.workunit.client.0.smithi090.stdout: "num_ops": 0 2021-05-12T12:08:53.483 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:08:53.490 INFO:tasks.workunit.client.0.smithi090.stdout:allowing recovery 2021-05-12T12:08:53.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2021-05-12T12:08:53.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:53.836 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:53.836 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084676+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.836 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.162528+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.837 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679024+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.837 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.683430+0000 31'100 61:141 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.837 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.308616+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.837 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681715+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.838 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.678417+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.838 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.083574+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.838 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086319+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.838 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.680039+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.839 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084491+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.839 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088175+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.839 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.310593+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.839 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:46.483841+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:46.483761+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.839 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.163435+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.840 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:50.293397+0000 31'100 62:128 [2,1,0] 2 [2,1,0] 2 31'100 2021-05-12T12:08:50.293336+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.840 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.089768+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.840 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:49.291955+0000 31'100 62:128 [2,1,0] 2 [2,1,0] 2 31'100 2021-05-12T12:08:49.291892+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.840 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086153+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.841 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:51.282236+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:51.282169+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.841 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681294+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.841 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:44.949907+0000 31'100 62:146 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:44.949847+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.841 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085787+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.841 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085136+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.842 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.308197+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.842 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841080+0000 31'100 62:141 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.842 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088770+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.842 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:47.964104+0000 31'100 62:148 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:47.964014+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.843 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.307136+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.843 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.162827+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.843 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.842131+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.843 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.678587+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.844 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085614+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.844 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682559+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.844 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679652+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.844 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086272+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.844 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.674466+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.845 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.082891+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.845 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.683139+0000 31'100 61:141 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.845 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086272+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.845 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681916+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.846 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.087893+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.846 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682048+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.846 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679651+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.846 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.163281+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.847 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086433+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.847 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088231+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.847 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.677512+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.847 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084553+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.848 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:45.922242+0000 31'100 62:146 [0,1,2] 0 [0,1,2] 0 31'100 2021-05-12T12:08:45.922169+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.848 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.674434+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.848 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085629+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.848 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean 2021-05-12T12:08:37.200182+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.848 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086758+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.849 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.677752+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.849 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.199559+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.849 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.203103+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.849 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.680426+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.849 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.674692+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.850 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086914+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.850 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.061004+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.850 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085984+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.850 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.677094+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.308202+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.202253+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085444+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.204762+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.851 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.679464+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.852 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.841700+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.852 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678128+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.852 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085224+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.852 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.205509+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.852 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean 2021-05-12T12:08:43.086625+0000 31'100 62:159 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.853 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.841928+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.853 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.199292+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.853 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.088061+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.853 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 2 0 0 0 0 861 0 0 17 17 active+clean 2021-05-12T12:08:37.203351+0000 54'101 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.853 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.084320+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.854 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086440+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.854 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085345+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.854 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.083397+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.854 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.203139+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.855 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean 2021-05-12T12:08:37.204131+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.855 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.163107+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.855 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 0 0 0 25 0 0 14 14 active+clean 2021-05-12T12:08:43.085059+0000 48'101 62:161 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.855 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085054+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.856 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678276+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.856 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086112+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.856 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086814+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.856 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.084738+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.856 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678131+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.857 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.087538+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.857 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086918+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.857 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:44.400740+0000 31'100 62:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:44.400687+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.857 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085811+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.857 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084785+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.858 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086721+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.858 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682125+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.858 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.674493+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.858 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679451+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.859 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682268+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.859 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.087715+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.859 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086592+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.859 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679275+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.860 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085458+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.860 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.059592+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.860 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean 2021-05-12T12:08:37.202043+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.860 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841490+0000 31'100 62:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.860 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679077+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.861 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682924+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.861 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.618377+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.861 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.837908+0000 31'100 62:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.861 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084135+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.862 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.199482+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.862 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.202749+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.862 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.082721+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.862 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.677952+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.862 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085795+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.863 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086101+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.863 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T12:08:43.361713+0000 31'100 62:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:43.361625+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.863 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084108+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.863 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:43.904146+0000 31'100 62:146 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:43.904073+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.864 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841285+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.864 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 4 0 0 0 0 1932 0 0 16 16 active+clean 2021-05-12T12:08:42.972577+0000 31'100 62:146 [0,1,2] 0 [0,1,2] 0 31'100 2021-05-12T12:08:42.972495+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.864 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682361+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.864 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:45.235970+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:45.235915+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.864 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:42.323015+0000 31'100 61:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:42.322962+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.865 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean 2021-05-12T12:08:43.081085+0000 31'100 62:159 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:53.865 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:53.865 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:53.866 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:53.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:53.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:08:53.970 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:53.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:53.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:08:54.101 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T12:08:54.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:08:54.234 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:08:54.243 INFO:tasks.workunit.client.0.smithi090.stdout:reading divergent objects 2021-05-12T12:08:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2021-05-12T12:08:54.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2021-05-12T12:08:54.589 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:08:54.590 INFO:tasks.workunit.client.0.smithi090.stdout:1.7f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084676+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.590 INFO:tasks.workunit.client.0.smithi090.stdout:1.7e 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.162528+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.590 INFO:tasks.workunit.client.0.smithi090.stdout:1.7d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679024+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7c 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.683430+0000 31'100 61:141 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.308616+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.7a 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681715+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.591 INFO:tasks.workunit.client.0.smithi090.stdout:1.79 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.678417+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.78 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.083574+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.77 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086319+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.76 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.680039+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.75 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084491+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.592 INFO:tasks.workunit.client.0.smithi090.stdout:1.74 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088175+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.73 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.310593+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.72 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:46.483841+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:46.483761+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.71 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.163435+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.593 INFO:tasks.workunit.client.0.smithi090.stdout:1.70 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:50.293397+0000 31'100 62:128 [2,1,0] 2 [2,1,0] 2 31'100 2021-05-12T12:08:50.293336+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.6f 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.089768+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.6e 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:49.291955+0000 31'100 62:128 [2,1,0] 2 [2,1,0] 2 31'100 2021-05-12T12:08:49.291892+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.6d 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086153+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.594 INFO:tasks.workunit.client.0.smithi090.stdout:1.6c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:51.282236+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:51.282169+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6b 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681294+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.6a 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:44.949907+0000 31'100 62:146 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:44.949847+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.69 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085787+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.595 INFO:tasks.workunit.client.0.smithi090.stdout:1.68 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085136+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.67 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.308197+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.66 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841080+0000 31'100 62:141 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.65 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088770+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.64 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:47.964104+0000 31'100 62:148 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:47.964014+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.596 INFO:tasks.workunit.client.0.smithi090.stdout:1.63 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.307136+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.62 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.162827+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.61 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.842131+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.60 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.678587+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.5f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085614+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.597 INFO:tasks.workunit.client.0.smithi090.stdout:1.5e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682559+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.5d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679652+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.5c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086272+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.5b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.674466+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.5a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.082891+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.598 INFO:tasks.workunit.client.0.smithi090.stdout:1.59 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.683139+0000 31'100 61:141 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.58 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086272+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.57 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.681916+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.56 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.087893+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.599 INFO:tasks.workunit.client.0.smithi090.stdout:1.55 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682048+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.54 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679651+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.53 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.163281+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.52 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086433+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.51 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.088231+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.600 INFO:tasks.workunit.client.0.smithi090.stdout:1.50 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.677512+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.4f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084553+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.4e 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:45.922242+0000 31'100 62:146 [0,1,2] 0 [0,1,2] 0 31'100 2021-05-12T12:08:45.922169+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.23 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.674434+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.22 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085629+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.601 INFO:tasks.workunit.client.0.smithi090.stdout:1.21 1 0 0 0 0 378 0 0 15 15 active+clean 2021-05-12T12:08:37.200182+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.20 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086758+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.1f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.677752+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.1e 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.199559+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.602 INFO:tasks.workunit.client.0.smithi090.stdout:1.1d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.203103+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.1c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.680426+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.1b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.674692+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.1a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086914+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.19 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.061004+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.603 INFO:tasks.workunit.client.0.smithi090.stdout:1.18 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085984+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.17 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.677094+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.16 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.308202+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.15 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.202253+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.604 INFO:tasks.workunit.client.0.smithi090.stdout:1.14 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085444+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.13 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.204762+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.12 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.679464+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.11 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.841700+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.10 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678128+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.605 INFO:tasks.workunit.client.0.smithi090.stdout:1.5 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085224+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.4 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.205509+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.3 1 0 0 0 0 378 0 0 10 10 active+clean 2021-05-12T12:08:43.086625+0000 31'100 62:159 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.2 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.841928+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.606 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.199292+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.1 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.088061+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.6 2 0 0 0 0 861 0 0 17 17 active+clean 2021-05-12T12:08:37.203351+0000 54'101 61:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.7 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.084320+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.8 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086440+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.607 INFO:tasks.workunit.client.0.smithi090.stdout:1.9 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085345+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.083397+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.b 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:37.203139+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.c 1 0 0 0 0 378 0 0 14 14 active+clean 2021-05-12T12:08:37.204131+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.608 INFO:tasks.workunit.client.0.smithi090.stdout:1.d 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.163107+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.e 1 0 0 0 0 25 0 0 14 14 active+clean 2021-05-12T12:08:43.085059+0000 48'101 62:161 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.f 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085054+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.24 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678276+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.25 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086112+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.609 INFO:tasks.workunit.client.0.smithi090.stdout:1.26 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086814+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.27 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.084738+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.28 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:42.678131+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.29 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.087538+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.2a 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.086918+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.610 INFO:tasks.workunit.client.0.smithi090.stdout:1.2b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:44.400740+0000 31'100 62:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:44.400687+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.2c 1 0 0 0 0 378 0 0 16 16 active+clean 2021-05-12T12:08:43.085811+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.2d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084785+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.2e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086721+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.611 INFO:tasks.workunit.client.0.smithi090.stdout:1.2f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682125+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.30 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.674493+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.31 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679451+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.32 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682268+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.612 INFO:tasks.workunit.client.0.smithi090.stdout:1.33 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.087715+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.34 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086592+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.35 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679275+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.36 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085458+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.37 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.059592+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.613 INFO:tasks.workunit.client.0.smithi090.stdout:1.38 1 0 0 0 0 377 0 0 10 10 active+clean 2021-05-12T12:08:37.202043+0000 31'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.39 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841490+0000 31'100 62:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.3a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.679077+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.3b 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682924+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.3c 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.618377+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.614 INFO:tasks.workunit.client.0.smithi090.stdout:1.3d 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.837908+0000 31'100 62:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.3e 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084135+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.3f 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.199482+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.40 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:37.202749+0000 31'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.615 INFO:tasks.workunit.client.0.smithi090.stdout:1.41 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.082721+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.42 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.677952+0000 31'100 61:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.43 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.085795+0000 31'100 62:123 [2,0,1] 2 [2,0,1] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.44 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.086101+0000 31'100 62:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.45 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T12:08:43.361713+0000 31'100 62:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:43.361625+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.616 INFO:tasks.workunit.client.0.smithi090.stdout:1.46 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:43.084108+0000 31'100 62:123 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.47 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:43.904146+0000 31'100 62:146 [0,2,1] 0 [0,2,1] 0 31'100 2021-05-12T12:08:43.904073+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.48 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.841285+0000 31'100 62:143 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.49 4 0 0 0 0 1932 0 0 16 16 active+clean 2021-05-12T12:08:42.972577+0000 31'100 62:146 [0,1,2] 0 [0,1,2] 0 31'100 2021-05-12T12:08:42.972495+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.617 INFO:tasks.workunit.client.0.smithi090.stdout:1.4a 1 0 0 0 0 377 0 0 16 16 active+clean 2021-05-12T12:08:42.682361+0000 31'100 61:143 [0,2,1] 0 [0,2,1] 0 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.4b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T12:08:45.235970+0000 31'100 62:128 [2,0,1] 2 [2,0,1] 2 31'100 2021-05-12T12:08:45.235915+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.4c 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T12:08:42.323015+0000 31'100 61:153 [1,0,2] 1 [1,0,2] 1 31'100 2021-05-12T12:08:42.322962+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi090.stdout:1.4d 1 0 0 0 0 377 0 0 12 12 active+clean 2021-05-12T12:08:43.081085+0000 31'100 62:159 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T12:06:09.248238+0000 0'0 2021-05-12T12:06:09.248238+0000 0 2021-05-12T12:08:54.618 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:08:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:08:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:54.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:08:54.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:54.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:08:54.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:08:54.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:08:54.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2021-05-12T12:08:54.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:08:54.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T12:08:54.776 INFO:tasks.workunit.client.0.smithi090.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T12:08:54.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T12:08:54.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T12:08:54.806 INFO:tasks.workunit.client.0.smithi090.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T12:08:54.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2021-05-12T12:08:54.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:54.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:08:54.859 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2021-05-12T12:06:36.798314+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.859 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 20 _merge_object_divergent_entries: keeping 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2021-05-12T12:06:36.798314+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.859 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:54.860 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 31'101 last_divergent_update: 31'101 2021-05-12T12:08:54.860 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T12:08:54.860 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:54.860 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 31'106 2021-05-12T12:08:54.861 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 31'101 (31'1) modify 1:d858c035:::existing_1:head by client.4534.0:1 2021-05-12T12:06:36.798314+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.861 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T12:08:54.861 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.345+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'84 2021-05-12T12:08:54.861 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b645e700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 31'105 (31'4) delete 1:70b3bf12:::existing_4:head by client.4546.0:1 2021-05-12T12:06:36.815692+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:54.861 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b645e700 20 _merge_object_divergent_entries: keeping 31'105 (31'4) delete 1:70b3bf12:::existing_4:head by client.4546.0:1 2021-05-12T12:06:36.815692+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:54.862 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b645e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:54.862 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b645e700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 31'105 last_divergent_update: 31'105 2021-05-12T12:08:54.862 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b645e700 10 _merge_object_divergent_entries: more recent entry found: 48'101 (31'4) modify 1:70b3bf12:::existing_4:head by client.4662.0:1 2021-05-12T12:07:36.147540+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T12:08:54.862 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 31'106 (31'3) delete 1:8881e6d8:::existing_3:head by client.4547.0:1 2021-05-12T12:06:36.817038+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:54.863 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 20 _merge_object_divergent_entries: keeping 31'106 (31'3) delete 1:8881e6d8:::existing_3:head by client.4547.0:1 2021-05-12T12:06:36.817038+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:54.863 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T12:08:54.863 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 31'106 last_divergent_update: 31'106 2021-05-12T12:08:54.863 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T12:08:54.864 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:54.864 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 31'106 2021-05-12T12:08:54.864 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 31'106 (31'3) delete 1:8881e6d8:::existing_3:head by client.4547.0:1 2021-05-12T12:06:36.817038+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T12:08:54.864 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T12:08:54.864 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.349+0000 7f43b7460700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'84 2021-05-12T12:08:54.865 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4540.0:1 2021-05-12T12:06:36.810086+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.865 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 20 _merge_object_divergent_entries: keeping 31'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4540.0:1 2021-05-12T12:06:36.810086+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.865 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:54.865 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 31'103 last_divergent_update: 31'103 2021-05-12T12:08:54.865 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T12:08:54.866 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.357+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T12:08:54.866 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 31'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4542.0:1 2021-05-12T12:06:36.812227+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.866 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 20 _merge_object_divergent_entries: keeping 31'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4542.0:1 2021-05-12T12:06:36.812227+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.866 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:54.870 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 31'104 last_divergent_update: 31'104 2021-05-12T12:08:54.871 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T12:08:54.871 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.358+0000 7f43b7c61700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T12:08:54.871 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2021-05-12T12:06:36.798327+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.871 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 20 _merge_object_divergent_entries: keeping 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2021-05-12T12:06:36.798327+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.871 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:08:54.872 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 31'102 last_divergent_update: 31'102 2021-05-12T12:08:54.872 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T12:08:54.872 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:08:54.872 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 31'106 2021-05-12T12:08:54.872 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 31'102 (31'2) modify 1:97db2a0f:::existing_2:head by client.4535.0:1 2021-05-12T12:06:36.798327+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:08:54.873 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T12:08:54.873 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:08:20.449+0000 7f43b8462700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'84 2021-05-12T12:08:54.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:54.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:08:54.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T12:08:54.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:08:54.984 INFO:tasks.workunit.client.0.smithi090.stdout:success 2021-05-12T12:08:54.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2021-05-12T12:08:54.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.29659.out 2021-05-12T12:08:54.985 INFO:tasks.workunit.client.0.smithi090.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.29659.out': No such file or directory 2021-05-12T12:08:54.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2021-05-12T12:08:54.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:08:54.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:08:55.557 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' does not exist 2021-05-12T12:08:55.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2021-05-12T12:08:55.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:55.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:55.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:55.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:55.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:55.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:55.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:08:55.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:08:55.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:08:55.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:08:55.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:55.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:55.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:55.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:55.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:55.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:55.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:08:55.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:08:55.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:08:55.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:08:55.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:08:55.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:08:55.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:55.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:08:55.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:08:55.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:55.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:08:55.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:08:55.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:08:55.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:08:55.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:55.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:55.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:08:55.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:08:55.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:08:55.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T12:08:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T12:08:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T12:08:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T12:08:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:08:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:08:55.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:08:55.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:08:55.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:08:55.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:08:55.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:08:55.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:08:55.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:08:55.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:08:55.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:08:55.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:08:55.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:08:55.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:08:55.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:08:55.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:55.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:08:55.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:08:55.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:08:55.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:08:55.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:08:55.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:08:55.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:08:55.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:55.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:55.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:08:55.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:08:55.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:08:55.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T12:08:55.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:08:55.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:55.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:55.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29659 2021-05-12T12:08:55.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:08:55.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:08:55.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:08:55.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T12:08:55.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2021-05-12T12:08:55.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2021-05-12T12:08:55.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2021-05-12T12:08:55.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2021-05-12T12:08:55.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2021-05-12T12:08:55.736 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2021-05-12T12:08:55.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2021-05-12T12:08:55.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2021-05-12T12:08:55.738 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:08:55.738 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:08:55.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2021-05-12T12:08:55.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T12:08:55.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:08:55.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:08:55.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:08:55.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T12:08:55.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T12:08:56.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:08:56.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:56.076 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:56.076 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:56.076 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:56.076 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:56.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:56.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:08:56.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:08:56.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:08:56.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:08:56.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:08:56.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:08:56.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:08:56.347 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:08:56.347 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:08:56.347 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:08:56.348 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:56.348 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:56.348 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:56.348 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:08:56.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:08:56.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get fsid 2021-05-12T12:08:56.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:08:56.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:08:56.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:08:56.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:08:56.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:08:56.443 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:08:56.443 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:08:56.443 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:08:56.443 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:08:56.444 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:56.444 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:56.444 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-mon.a.asok 2021-05-12T12:08:56.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:08:56.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29659/ceph-mon.a.asok config get mon_host 2021-05-12T12:08:56.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2021-05-12T12:08:56.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T12:08:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:08:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:08:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:08:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T12:08:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:08:56.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:08:56.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:56.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:56.998 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:56.998 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:56.998 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:56.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:56.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:08:56.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:08:57.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:08:57.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2021-05-12T12:08:57.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:08:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:08:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:08:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:08:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:08:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:08:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:08:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:08:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:08:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:08:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:08:57.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:08:57.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:08:57.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:08:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:08:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:08:57.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:08:57.032 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:08:57.032 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:08:57.032 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:08:57.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:57.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:08:57.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:08:57.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:08:57.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:08:57.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:08:57.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:08:57.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:08:57.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:08:57.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:08:57.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:08:57.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:08:57.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:08:57.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:08:57.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:08:57.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 01e065f1-a569-44af-90fd-e9233051a6cf' 2021-05-12T12:08:57.037 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:08:57.037 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:08:57.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBZxZtgIrm6AhAAsWTWptMrDRLYiEVCgYPulQ== 2021-05-12T12:08:57.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBZxZtgIrm6AhAAsWTWptMrDRLYiEVCgYPulQ=="}' 2021-05-12T12:08:57.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 01e065f1-a569-44af-90fd-e9233051a6cf -i td/divergent-priors/0/new.json 2021-05-12T12:08:57.377 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:08:57.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T12:08:57.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZxZtgIrm6AhAAsWTWptMrDRLYiEVCgYPulQ== --osd-uuid 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:08:57.418 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.415+0000 7f78771fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:57.421 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.417+0000 7f78771fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:57.422 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.418+0000 7f78771fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:08:57.423 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.419+0000 7f78771fd0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:57.423 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.419+0000 7f78771fd0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:57.423 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.419+0000 7f78771fd0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T12:08:57.460 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:08:57.456+0000 7f78771fd0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T12:09:01.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T12:09:01.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:09:01.719 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:09:01.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:09:01.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:09:02.196 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:09:02.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:09:02.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:02.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:09:02.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:09:02.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:09:02.224 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:02.222+0000 7f923218c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:02.227 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:02.224+0000 7f923218c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:02.227 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:02.225+0000 7f923218c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:02.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:09:02.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:09:02.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:09:02.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:09:02.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:09:02.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:02.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:09:02.563 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:09:02.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:02.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:09:02.743 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:02.741+0000 7f923218c0c0 -1 Falling back to public interface 2021-05-12T12:09:02.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:03.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:03.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:03.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:09:03.942 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:09:03.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:03.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:09:04.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:04.923 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:04.920+0000 7f923218c0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:09:05.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:05.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:05.312 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:09:05.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:09:05.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:05.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:09:05.696 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/69683,v1:127.0.0.1:6803/69683] [v2:127.0.0.1:6804/69683,v1:127.0.0.1:6805/69683] exists,up 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:09:05.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:09:05.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:09:05.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:09:05.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:05.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2021-05-12T12:09:05.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:09:05.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:09:05.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:09:05.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:09:05.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:09:05.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:09:05.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:09:05.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:09:05.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:09:05.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:09:05.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:09:05.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:09:05.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:09:05.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:09:05.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:09:05.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:09:05.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:09:05.705 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:09:05.705 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:09:05.706 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:09:05.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:09:05.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:09:05.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:09:05.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:09:05.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:09:05.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:09:05.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:09:05.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:09:05.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:09:05.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:05.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:09:05.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:09:05.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:09:05.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:09:05.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:09:05.709 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:09:05.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 219c9c7a-4f27-4908-a6a5-bc27283f6f3a' 2021-05-12T12:09:05.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:09:05.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBhxZtgWZ6tKhAAxuj++8hZaS5E+c1XiD+s7Q== 2021-05-12T12:09:05.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBhxZtgWZ6tKhAAxuj++8hZaS5E+c1XiD+s7Q=="}' 2021-05-12T12:09:05.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 219c9c7a-4f27-4908-a6a5-bc27283f6f3a -i td/divergent-priors/1/new.json 2021-05-12T12:09:06.180 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:09:06.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T12:09:06.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBhxZtgWZ6tKhAAxuj++8hZaS5E+c1XiD+s7Q== --osd-uuid 219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:09:06.222 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.219+0000 7f32489480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:06.224 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.221+0000 7f32489480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:06.225 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.222+0000 7f32489480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:06.226 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.224+0000 7f32489480c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:09:06.226 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.224+0000 7f32489480c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:09:06.227 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.224+0000 7f32489480c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T12:09:06.320 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:06.314+0000 7f32489480c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T12:09:10.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T12:09:10.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:09:10.897 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:09:10.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:09:10.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:09:11.384 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:09:11.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:09:11.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:09:11.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:11.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:09:11.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:09:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:11.410+0000 7f16201e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:11.429 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:11.427+0000 7f16201e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:11.431 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:11.428+0000 7f16201e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:11.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:09:11.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:09:11.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:09:11.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:09:11.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:09:11.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:11.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:09:11.754 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:09:11.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:11.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:09:11.947 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:11.945+0000 7f16201e30c0 -1 Falling back to public interface 2021-05-12T12:09:12.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:13.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:13.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:13.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:09:13.117 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:09:13.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:13.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:09:13.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:14.132 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:14.130+0000 7f16201e30c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:09:14.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:14.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:14.490 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:09:14.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:09:14.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:14.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:09:14.874 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/71068,v1:127.0.0.1:6811/71068] [v2:127.0.0.1:6812/71068,v1:127.0.0.1:6813/71068] exists,up 219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:09:14.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:09:14.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:09:14.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:09:14.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:14.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2021-05-12T12:09:14.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T12:09:14.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:09:14.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:09:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:09:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:09:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:09:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:09:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:09:14.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:09:14.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:09:14.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:09:14.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:09:14.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:09:14.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:09:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:09:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:09:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:09:14.884 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:09:14.884 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:09:14.884 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:09:14.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:09:14.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:09:14.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:09:14.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:09:14.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:09:14.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:09:14.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:09:14.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:09:14.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:09:14.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:14.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:09:14.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:09:14.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:09:14.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:09:14.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:09:14.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a' 2021-05-12T12:09:14.888 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:09:14.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:09:14.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBqxZtg4yVONRAAPuZjxC4SjCrbqx1B7XJpBw== 2021-05-12T12:09:14.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBqxZtg4yVONRAAPuZjxC4SjCrbqx1B7XJpBw=="}' 2021-05-12T12:09:14.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a -i td/divergent-priors/2/new.json 2021-05-12T12:09:15.468 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:09:15.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T12:09:15.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBqxZtg4yVONRAAPuZjxC4SjCrbqx1B7XJpBw== --osd-uuid 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:09:15.509 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.506+0000 7f9c9557e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:15.511 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.508+0000 7f9c9557e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:15.511 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.509+0000 7f9c9557e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:15.512 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.509+0000 7f9c9557e0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:09:15.513 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.509+0000 7f9c9557e0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:09:15.513 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.509+0000 7f9c9557e0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T12:09:15.597 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:15.595+0000 7f9c9557e0c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T12:09:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T12:09:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:09:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:09:20.433 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:09:20.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:09:20.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:09:20.915 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:09:20.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:09:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:09:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:09:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:09:20.942 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:20.940+0000 7ff3193000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:20.944 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:20.942+0000 7ff3193000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:20.946 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:20.943+0000 7ff3193000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:09:21.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:09:21.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:09:21.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:09:21.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:09:21.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:09:21.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:21.284 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:09:21.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:09:21.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:21.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:21.462 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:21.460+0000 7ff3193000c0 -1 Falling back to public interface 2021-05-12T12:09:21.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:22.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:22.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:22.660 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:09:22.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:09:22.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:22.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:23.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:09:23.683 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:09:23.680+0000 7ff3193000c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:09:24.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:09:24.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:09:24.044 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:09:24.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:09:24.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:09:24.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:09:24.441 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/72453,v1:127.0.0.1:6819/72453] [v2:127.0.0.1:6820/72453,v1:127.0.0.1:6821/72453] exists,up 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:09:24.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:09:24.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:09:24.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:09:24.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2021-05-12T12:09:25.252 INFO:tasks.workunit.client.0.smithi090.stderr:noout is set 2021-05-12T12:09:25.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2021-05-12T12:09:25.962 INFO:tasks.workunit.client.0.smithi090.stderr:noin is set 2021-05-12T12:09:25.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2021-05-12T12:09:26.673 INFO:tasks.workunit.client.0.smithi090.stderr:nodown is set 2021-05-12T12:09:26.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2021-05-12T12:09:26.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: local pool_name=test 2021-05-12T12:09:26.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: shift 2021-05-12T12:09:26.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: create_ec_pool: local allow_overwrites=true 2021-05-12T12:09:26.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: shift 2021-05-12T12:09:26.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-05-12T12:09:27.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: create_pool test 1 1 erasure myprofile 2021-05-12T12:09:27.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-05-12T12:09:28.166 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' already exists 2021-05-12T12:09:28.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:09:29.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: create_ec_pool: '[' true = true ']' 2021-05-12T12:09:29.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2021-05-12T12:09:29.879 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 allow_ec_overwrites to true 2021-05-12T12:09:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: wait_for_clean 2021-05-12T12:09:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:09:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:09:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:09:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:09:29.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:09:29.890 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:09:29.890 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:09:29.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:09:29.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:09:29.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:09:29.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:09:29.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:09:29.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:09:29.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:29.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:30.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:30.343 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:09:30.343 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:09:30.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:30.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:30.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:30.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T12:09:30.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T12:09:30.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T12:09:30.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:30.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:30.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705671 2021-05-12T12:09:30.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705671 2021-05-12T12:09:30.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-38654705671' 2021-05-12T12:09:30.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:30.609 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:30.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542148 2021-05-12T12:09:30.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542148 2021-05-12T12:09:30.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-38654705671 2-60129542148' 2021-05-12T12:09:30.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:30.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T12:09:30.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:30.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:30.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T12:09:30.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:30.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T12:09:30.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T12:09:30.745 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836487 2021-05-12T12:09:30.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:31.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T12:09:31.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:32.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:32.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:32.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836487 2021-05-12T12:09:32.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:32.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705671 2021-05-12T12:09:32.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:32.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:32.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705671 2021-05-12T12:09:32.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:32.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705671 2021-05-12T12:09:32.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2021-05-12T12:09:32.475 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705671 2021-05-12T12:09:32.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:32.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705671 -lt 38654705671 2021-05-12T12:09:32.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:32.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2021-05-12T12:09:32.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:32.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:32.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542148 2021-05-12T12:09:32.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542148 2021-05-12T12:09:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2021-05-12T12:09:32.841 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542148 2021-05-12T12:09:32.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:33.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542148 -lt 60129542148 2021-05-12T12:09:33.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:09:33.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:33.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:33.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:09:33.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:09:33.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:09:33.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:09:33.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:09:33.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:09:33.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:09:33.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:09:34.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:09:34.043 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:09:34.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:34.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:34.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:09:34.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:09:34.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:09:34.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: create_ec_pool: return 0 2021-05-12T12:09:34.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2021-05-12T12:09:34.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:34.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:34.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:34.879 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:09:34.880 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:09:34.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:34.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:34.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:35.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T12:09:35.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T12:09:35.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T12:09:35.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:35.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:35.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705673 2021-05-12T12:09:35.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705673 2021-05-12T12:09:35.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-38654705673' 2021-05-12T12:09:35.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:35.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:35.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542151 2021-05-12T12:09:35.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542151 2021-05-12T12:09:35.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-38654705673 2-60129542151' 2021-05-12T12:09:35.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:35.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T12:09:35.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:35.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:35.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T12:09:35.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:35.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T12:09:35.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T12:09:35.282 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836490 2021-05-12T12:09:35.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:35.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T12:09:35.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:35.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705673 2021-05-12T12:09:35.699 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:35.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:35.700 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705673 2021-05-12T12:09:35.700 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:35.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705673 2021-05-12T12:09:35.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2021-05-12T12:09:35.701 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705673 2021-05-12T12:09:35.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:36.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705674 -lt 38654705673 2021-05-12T12:09:36.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:36.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542151 2021-05-12T12:09:36.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:36.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:36.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542151 2021-05-12T12:09:36.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:36.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542151 2021-05-12T12:09:36.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2021-05-12T12:09:36.069 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542151 2021-05-12T12:09:36.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:36.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542151 -lt 60129542151 2021-05-12T12:09:36.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2021-05-12T12:09:36.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:09:36.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:09:36.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:09:36.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:09:36.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:09:36.438 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:09:36.438 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:09:36.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:09:36.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:09:36.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:09:36.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:09:36.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:09:36.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:09:36.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:09:36.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:09:36.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:09:36.912 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:09:36.912 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:09:36.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:09:36.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:36.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:09:37.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836493 2021-05-12T12:09:37.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836493 2021-05-12T12:09:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493' 2021-05-12T12:09:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:37.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:09:37.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705676 2021-05-12T12:09:37.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705676 2021-05-12T12:09:37.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2021-05-12T12:09:37.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:09:37.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:09:37.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542153 2021-05-12T12:09:37.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542153 2021-05-12T12:09:37.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542153' 2021-05-12T12:09:37.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:37.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-05-12T12:09:37.312 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:37.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:09:37.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836493 2021-05-12T12:09:37.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:37.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836493 2021-05-12T12:09:37.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-05-12T12:09:37.314 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836493 2021-05-12T12:09:37.315 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836493 2021-05-12T12:09:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:09:38.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:09:38.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:09:39.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836493 -lt 21474836493 2021-05-12T12:09:39.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:39.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705676 2021-05-12T12:09:39.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:39.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:09:39.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705676 2021-05-12T12:09:39.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:39.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705676 2021-05-12T12:09:39.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2021-05-12T12:09:39.052 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705676 2021-05-12T12:09:39.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:09:39.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705676 -lt 38654705676 2021-05-12T12:09:39.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:09:39.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542153 2021-05-12T12:09:39.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:09:39.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:09:39.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542153 2021-05-12T12:09:39.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:09:39.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542153 2021-05-12T12:09:39.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2021-05-12T12:09:39.418 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542153 2021-05-12T12:09:39.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:09:39.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542153 -lt 60129542153 2021-05-12T12:09:39.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:09:39.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:39.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:40.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:09:40.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:09:40.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:09:40.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:09:40.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:09:40.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:09:40.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:09:40.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:09:40.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:09:40.631 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:09:40.631 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:09:40.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:09:41.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:09:41.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:09:41.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:09:41.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2021-05-12T12:09:41.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2021-05-12T12:09:41.445 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:09:41.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2021-05-12T12:09:41.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2021-05-12T12:09:41.455 INFO:tasks.workunit.client.0.smithi090.stdout:primary and soon to be divergent is 1 2021-05-12T12:09:41.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:09:41.797 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:09:41.798 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:09:29.044927+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:09:41.798 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:09:41.798 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:09:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:09:41.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2021-05-12T12:09:41.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:41.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2021-05-12T12:09:41.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2021-05-12T12:09:41.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:41.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2021-05-12T12:09:41.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2021-05-12T12:09:41.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T12:09:41.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2021-05-12T12:09:41.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2021-05-12T12:09:41.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2021-05-12T12:09:41.812 INFO:tasks.workunit.client.0.smithi090.stdout:writing initial objects 2021-05-12T12:09:41.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2021-05-12T12:09:41.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2021-05-12T12:09:41.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:41.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2021-05-12T12:09:42.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2021-05-12T12:09:42.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2021-05-12T12:09:42.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2021-05-12T12:09:42.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:42.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2021-05-12T12:09:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2021-05-12T12:09:43.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2021-05-12T12:09:43.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2021-05-12T12:09:43.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2021-05-12T12:09:43.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:43.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2021-05-12T12:09:44.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2021-05-12T12:09:44.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2021-05-12T12:09:44.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2021-05-12T12:09:44.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2021-05-12T12:09:44.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:44.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2021-05-12T12:09:45.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2021-05-12T12:09:45.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2021-05-12T12:09:45.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2021-05-12T12:09:45.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:45.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2021-05-12T12:09:46.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2021-05-12T12:09:46.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2021-05-12T12:09:46.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2021-05-12T12:09:46.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2021-05-12T12:09:46.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:46.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2021-05-12T12:09:47.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:47.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2021-05-12T12:09:47.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:47.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2021-05-12T12:09:47.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:47.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2021-05-12T12:09:47.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:47.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2021-05-12T12:09:48.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:48.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2021-05-12T12:09:48.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:48.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2021-05-12T12:09:48.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:48.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2021-05-12T12:09:48.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:48.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2021-05-12T12:09:48.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:48.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2021-05-12T12:09:49.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:49.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2021-05-12T12:09:49.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:49.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2021-05-12T12:09:49.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:49.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2021-05-12T12:09:49.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:49.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2021-05-12T12:09:49.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:49.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2021-05-12T12:09:50.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:50.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2021-05-12T12:09:50.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:50.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2021-05-12T12:09:50.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:50.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2021-05-12T12:09:50.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:50.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2021-05-12T12:09:51.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:51.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2021-05-12T12:09:51.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:51.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2021-05-12T12:09:51.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:51.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2021-05-12T12:09:51.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:51.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2021-05-12T12:09:51.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:51.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2021-05-12T12:09:52.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:52.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2021-05-12T12:09:52.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:52.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2021-05-12T12:09:52.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:52.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2021-05-12T12:09:52.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:52.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2021-05-12T12:09:52.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:52.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2021-05-12T12:09:53.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:53.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2021-05-12T12:09:53.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:53.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2021-05-12T12:09:53.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:53.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2021-05-12T12:09:53.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:53.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2021-05-12T12:09:54.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:54.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2021-05-12T12:09:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:54.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2021-05-12T12:09:54.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:54.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2021-05-12T12:09:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:54.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2021-05-12T12:09:54.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:54.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2021-05-12T12:09:55.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:55.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2021-05-12T12:09:55.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:55.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2021-05-12T12:09:55.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:55.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2021-05-12T12:09:55.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:55.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2021-05-12T12:09:55.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:55.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2021-05-12T12:09:56.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:56.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2021-05-12T12:09:56.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:56.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2021-05-12T12:09:56.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:56.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2021-05-12T12:09:56.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:56.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2021-05-12T12:09:57.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:57.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2021-05-12T12:09:57.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:57.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2021-05-12T12:09:57.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:57.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2021-05-12T12:09:57.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:57.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2021-05-12T12:09:57.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:57.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2021-05-12T12:09:58.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:58.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2021-05-12T12:09:58.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:58.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2021-05-12T12:09:58.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:58.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2021-05-12T12:09:59.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:59.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2021-05-12T12:09:59.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:59.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2021-05-12T12:09:59.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:59.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2021-05-12T12:09:59.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:59.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2021-05-12T12:09:59.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:09:59.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2021-05-12T12:10:00.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:00.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2021-05-12T12:10:00.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:00.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2021-05-12T12:10:00.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:00.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2021-05-12T12:10:00.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:00.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2021-05-12T12:10:01.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:01.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2021-05-12T12:10:01.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:01.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2021-05-12T12:10:01.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:01.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2021-05-12T12:10:01.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:01.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2021-05-12T12:10:01.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:01.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2021-05-12T12:10:02.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:02.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2021-05-12T12:10:02.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:02.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2021-05-12T12:10:02.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:02.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2021-05-12T12:10:02.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:02.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2021-05-12T12:10:03.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:03.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2021-05-12T12:10:03.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:03.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T12:10:03.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T12:10:03.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T12:10:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:10:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2021-05-12T12:10:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:10:03.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:10:03.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:10:03.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:10:03.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:10:03.626 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:10:03.627 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:10:03.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:10:03.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:10:03.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:10:03.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:10:03.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:10:03.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:10:03.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:10:03.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:10:04.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:10:04.058 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:10:04.062 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:10:04.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:10:04.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:04.064 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:10:04.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836500 2021-05-12T12:10:04.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836500 2021-05-12T12:10:04.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500' 2021-05-12T12:10:04.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:04.218 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:10:04.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705683 2021-05-12T12:10:04.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705683 2021-05-12T12:10:04.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500 1-38654705683' 2021-05-12T12:10:04.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:04.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:10:04.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542160 2021-05-12T12:10:04.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542160 2021-05-12T12:10:04.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500 1-38654705683 2-60129542160' 2021-05-12T12:10:04.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:04.478 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-05-12T12:10:04.478 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:04.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:10:04.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836500 2021-05-12T12:10:04.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:04.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836500 2021-05-12T12:10:04.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-05-12T12:10:04.483 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836500 2021-05-12T12:10:04.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:04.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836498 -lt 21474836500 2021-05-12T12:10:04.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:10:05.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:10:05.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:06.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836500 -lt 21474836500 2021-05-12T12:10:06.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:06.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705683 2021-05-12T12:10:06.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:06.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:10:06.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705683 2021-05-12T12:10:06.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:06.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705683 2021-05-12T12:10:06.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2021-05-12T12:10:06.220 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705683 2021-05-12T12:10:06.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:10:06.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705684 -lt 38654705683 2021-05-12T12:10:06.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:06.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542160 2021-05-12T12:10:06.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:06.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:10:06.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542160 2021-05-12T12:10:06.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:06.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542160 2021-05-12T12:10:06.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2021-05-12T12:10:06.584 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542160 2021-05-12T12:10:06.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:10:06.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542161 -lt 60129542160 2021-05-12T12:10:06.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:10:06.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:06.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:07.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:10:07.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:07.463 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:07.463 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:07.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:07.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:07.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:07.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:07.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:10:07.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:07.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:07.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:08.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:10:08.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:10:08.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:10:08.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2021-05-12T12:10:08.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T12:10:08.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T12:10:08.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T12:10:08.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T12:10:08.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2021-05-12T12:10:08.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2021-05-12T12:10:08.686 INFO:tasks.workunit.client.0.smithi090.stdout:blackholing osds 0 2 2021-05-12T12:10:08.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:09.046 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:09.046 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T12:09:29.044927+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:10:09.047 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:10:09.047 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:09.048 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:10:09.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:09.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T12:10:09.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:10:09.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:10:09.059 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:10:09.060 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:09.060 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:10:09.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:10:09.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:10:09.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T12:10:09.149 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:10:09.150 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:10:09.150 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:10:09.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:09.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2021-05-12T12:10:09.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:10:09.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:10:09.161 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:10:09.161 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:09.161 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:10:09.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:10:09.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:10:09.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok config set objectstore_blackhole 1 2021-05-12T12:10:09.250 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:10:09.251 INFO:tasks.workunit.client.0.smithi090.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T12:10:09.251 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:10:09.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2021-05-12T12:10:09.259 INFO:tasks.workunit.client.0.smithi090.stdout:writing divergent object 2021-05-12T12:10:09.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2021-05-12T12:10:09.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2021-05-12T12:10:10.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2021-05-12T12:10:10.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T12:10:10.465 INFO:tasks.workunit.client.0.smithi090.stdout:created pool test snap snap1 2021-05-12T12:10:10.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2021-05-12T12:10:10.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2021-05-12T12:10:10.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T12:10:20.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2021-05-12T12:10:20.472 INFO:tasks.workunit.client.0.smithi090.stderr:rados: no process found 2021-05-12T12:10:20.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2021-05-12T12:10:20.473 INFO:tasks.workunit.client.0.smithi090.stdout:killing all the osds 2021-05-12T12:10:20.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:20.827 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:20.828 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2021-05-12T12:09:29.044927+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:10:20.828 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:10:20.828 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:20.829 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:10:20.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2021-05-12T12:10:20.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:10:20.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:10:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:10:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:10:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:10:20.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:10:20.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:20.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2021-05-12T12:10:21.481 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:10:21.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:21.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2021-05-12T12:10:21.991 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:10:22.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T12:10:22.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2021-05-12T12:10:22.593 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:10:22.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:22.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2021-05-12T12:10:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already out. 2021-05-12T12:10:23.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:23.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2021-05-12T12:10:23.608 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already out. 2021-05-12T12:10:23.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2021-05-12T12:10:23.618 INFO:tasks.workunit.client.0.smithi090.stdout:bringing up non_divergent 0 2 2021-05-12T12:10:23.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:23.956 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:23.956 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2021-05-12T12:09:29.044927+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:10:23.956 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:10:23.957 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:23.958 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:10:23.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:23.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T12:10:23.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:10:23.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:10:23.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:10:23.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:10:23.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:10:23.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:10:23.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:10:23.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:10:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:10:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:10:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:10:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:10:23.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:10:23.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:10:23.975 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:10:23.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:10:23.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:10:23.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:10:23.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:10:23.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:10:23.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:10:23.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:23.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:10:23.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:10:23.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:10:23.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:10:23.978 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:10:23.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:23.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:10:23.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:10:23.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:10:23.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:10:23.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:10:23.998 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:23.995+0000 7f447a85c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:24.000 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:23.997+0000 7f447a85c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:24.001 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:23.999+0000 7f447a85c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:24.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:10:24.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:10:24.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:10:24.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:10:24.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:10:24.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:24.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:10:24.336 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:10:24.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:24.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:24.518 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:24.515+0000 7f447a85c0c0 -1 Falling back to public interface 2021-05-12T12:10:24.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:25.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:25.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:25.703 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:10:25.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:10:25.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:25.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:26.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:27.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:27.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:27.075 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:10:27.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:10:27.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:27.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:27.181 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:27.178+0000 7f447a85c0c0 -1 osd.0 28 log_to_monitors {default=true} 2021-05-12T12:10:27.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:28.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:28.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:28.454 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:10:28.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:10:28.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:28.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:10:28.824 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/76874,v1:127.0.0.1:6803/76874] [v2:127.0.0.1:6804/76874,v1:127.0.0.1:6805/76874] exists,up 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:10:28.828 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:10:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:10:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:10:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T12:10:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:10:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:10:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:10:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:10:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:10:28.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:10:28.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:10:28.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:10:28.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:10:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:10:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:10:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:10:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:10:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:10:28.833 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:10:28.833 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:10:28.833 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:10:28.833 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:10:28.834 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:10:28.834 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:10:28.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:10:28.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:10:28.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:10:28.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:10:28.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:10:28.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:10:28.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:10:28.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:28.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:10:28.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:10:28.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:10:28.837 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:10:28.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:10:28.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:10:28.838 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:10:28.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:10:28.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:10:28.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:10:28.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:10:28.855 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:28.852+0000 7f526890c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:28.856 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:28.854+0000 7f526890c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:28.858 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:28.855+0000 7f526890c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:10:29.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:10:29.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:10:29.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:10:29.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:10:29.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:10:29.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:29.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:10:29.196 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:10:29.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:29.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:29.373 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:29.371+0000 7f526890c0c0 -1 Falling back to public interface 2021-05-12T12:10:29.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:30.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:30.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:30.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:10:30.582 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:10:30.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:30.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:30.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:31.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:31.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:31.949 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:10:31.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:10:31.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:31.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:32.091 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:10:32.088+0000 7f526890c0c0 -1 osd.2 28 log_to_monitors {default=true} 2021-05-12T12:10:32.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:10:33.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:10:33.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:10:33.339 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:10:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:10:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:10:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:10:33.731 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [14,28) [v2:127.0.0.1:6810/77457,v1:127.0.0.1:6811/77457] [v2:127.0.0.1:6812/77457,v1:127.0.0.1:6813/77457] exists,up 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:10:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:10:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:10:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:10:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2021-05-12T12:10:34.209 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already in. 2021-05-12T12:10:34.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:10:34.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2021-05-12T12:10:34.915 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already in. 2021-05-12T12:10:34.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2021-05-12T12:10:39.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2021-05-12T12:10:39.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2021-05-12T12:10:39.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2021-05-12T12:10:39.925 INFO:tasks.workunit.client.0.smithi090.stdout:writing non-divergent object existing_4 2021-05-12T12:10:39.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:10:40.273 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:10:40.274 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T12:10:35.318684+0000 28'100 43:124 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:10:40.274 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:10:40.274 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:10:40.275 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:10:40.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T12:10:40.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:10:40.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2021-05-12T12:10:40.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:10:40.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:10:40.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:10:40.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:10:40.394 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:10:40.394 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:10:40.394 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:10:40.395 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:10:40.395 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:10:40.395 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:10:40.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:10:40.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:10:40.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:10:40.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:10:40.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:10:40.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:10:40.813 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:10:40.813 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:10:40.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:10:40.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:40.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:10:40.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822661 2021-05-12T12:10:40.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822661 2021-05-12T12:10:40.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-154618822661' 2021-05-12T12:10:40.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:40.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:10:41.060 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:10:41.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:10:41.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:10:41.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:10:41.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:10:41.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:10:41.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=163208757252 2021-05-12T12:10:41.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 163208757252 2021-05-12T12:10:41.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-154618822661 2-163208757252' 2021-05-12T12:10:41.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:41.198 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822661 2021-05-12T12:10:41.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:41.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:10:41.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-154618822661 2021-05-12T12:10:41.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:41.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822661 2021-05-12T12:10:41.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2021-05-12T12:10:41.201 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 154618822661 2021-05-12T12:10:41.201 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:41.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822659 -lt 154618822661 2021-05-12T12:10:41.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:10:42.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:10:42.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:10:42.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822661 -lt 154618822661 2021-05-12T12:10:42.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:10:42.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-163208757252 2021-05-12T12:10:42.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:10:42.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:10:42.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-163208757252 2021-05-12T12:10:42.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:10:42.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=163208757252 2021-05-12T12:10:42.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2021-05-12T12:10:42.936 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 163208757252 2021-05-12T12:10:42.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:10:43.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 163208757252 -lt 163208757252 2021-05-12T12:10:43.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:10:43.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:43.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:43.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:10:43.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:43.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:43.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:43.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:43.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:43.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:43.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:44.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:44.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:44.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:44.169 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:44.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:44.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:10:44.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:10:44.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:10:44.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:44.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:10:44.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:44.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:44.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:44.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:44.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:44.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:44.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:44.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:45.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:45.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:45.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:45.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:45.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:45.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:45.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:45.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:45.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:45.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:45.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:45.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:45.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:46.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:46.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:46.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:10:46.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:46.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:10:46.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:46.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:46.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:46.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:46.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:46.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:46.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:46.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:46.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:46.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:46.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:46.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:47.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:47.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:47.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:47.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:47.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:47.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:47.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:47.088 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:47.089 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:47.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:47.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:47.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:10:47.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:47.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:10:47.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:47.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:47.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:47.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:47.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:47.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:47.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:47.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:48.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:48.373 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:48.373 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:48.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:48.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:48.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:48.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:48.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:48.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:48.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:48.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:48.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:48.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:49.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:49.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:49.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:10:49.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:49.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:10:50.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:50.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:50.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:50.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:50.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:50.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:50.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:50.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:50.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:50.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:50.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:50.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:50.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:50.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:50.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:50.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:50.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:50.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:50.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:50.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:50.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:51.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:51.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:51.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:10:51.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:51.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:10:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:53.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:53.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:53.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:53.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:53.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:53.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:53.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:53.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:53.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:53.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:53.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:53.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:53.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:53.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:53.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:53.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:53.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:53.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:53.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:54.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:10:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:54.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:10:57.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:10:57.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:10:57.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:10:57.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:10:57.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:10:57.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:10:57.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:10:57.569 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:10:57.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:10:57.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:10:57.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:10:57.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:10:58.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:10:58.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:10:58.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:10:58.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:10:58.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:10:58.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:10:58.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:10:58.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:10:58.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:10:58.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:10:58.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:11:05.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:05.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:05.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:05.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:05.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:05.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:05.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:05.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:05.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:05.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:05.650 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:05.650 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:06.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:06.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:06.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:06.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:06.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:06.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:06.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:06.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:06.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:06.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:06.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:06.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:11:06.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:06.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:11:13.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:13.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:13.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:13.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:13.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:13.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:13.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:13.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:14.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:14.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:14.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:14.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:14.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:14.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:14.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:14.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:14.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:14.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:15.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:15.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:15.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:11:15.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:11:15.682 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:11:15.683 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "05499b63-48ca-400c-9b23-2220766a714e", 2021-05-12T12:11:15.683 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:15.683 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:11:15.684 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:11:15.678052+0000", 2021-05-12T12:11:15.684 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:11:15.685 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:11:15.685 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:11:15.686 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:11:15.686 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:11:15.687 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:11:15.687 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:15.687 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:11:15.688 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:15.688 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.689 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:15.689 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.690 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:11:15.690 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.691 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.691 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:15.692 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.692 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:11:15.692 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:15.693 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:15.693 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:11:15.694 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:11:15.694 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.695 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:11:15.695 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:15.695 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.696 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:11:15.696 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:15.697 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:15.697 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:11:15.697 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:15.698 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.698 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:15.699 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.699 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:11:15.700 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.700 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.700 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:15.701 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.701 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:11:15.702 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:15.702 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:15.703 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:11:15.703 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:15.703 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.704 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:15.704 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.705 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-05-12T12:11:15.705 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.706 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.706 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:15.706 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.707 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:11:15.707 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:15.708 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:15.708 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:11:15.709 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:15.709 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.709 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:15.710 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.710 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:11:15.711 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.711 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.712 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:11:15.712 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.713 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.713 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:15.713 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.714 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.714 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:11:15.715 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.715 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:11:15.716 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:11:15.716 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:11:15.716 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:11:15.717 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:11:15.717 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:08:55.755262Z", 2021-05-12T12:11:15.718 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:08:55.755262Z", 2021-05-12T12:11:15.718 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:11:15.719 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:11:15.719 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:11:15.719 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:11:15.720 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:11:15.720 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:11:15.721 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:11:15.721 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:11:15.721 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:11:15.722 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:11:15.722 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:11:15.723 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:11:15.723 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:11:15.724 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:11:15.724 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:11:15.724 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:11:15.725 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.725 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:11:15.726 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.726 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:11:15.726 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.727 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:11:15.727 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:11:15.727 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:15.728 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.728 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.728 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.729 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:11:15.729 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:11:15.729 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.729 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.729 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.730 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:11:15.730 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:11:15.730 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:11:15.730 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:11:15.731 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:11:15.731 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.731 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.731 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.731 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:11:15.732 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:11:15.732 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.732 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:11:15.732 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 43, 2021-05-12T12:11:15.733 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:11:15.733 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:08:56.622276+0000", 2021-05-12T12:11:15.733 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:10:34.975460+0000", 2021-05-12T12:11:15.733 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:10:32.443129+0000", 2021-05-12T12:11:15.733 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:10:34.503488+0000", 2021-05-12T12:11:15.734 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:11:15.734 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:11:15.734 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:11:15.734 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:11:15.735 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:11:15.735 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:11:15.735 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:11:15.735 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:11:15.736 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:11:15.736 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:11:15.736 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.736 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 10, 2021-05-12T12:11:15.736 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:11:15.737 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:11:15.737 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:11:15.737 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:11:15.737 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:11:15.737 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:11:15.738 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:11:15.738 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:11:15.738 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:11:15.738 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:11:15.739 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.739 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:11:15.739 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:11:15.739 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:09:27.835639+0000", 2021-05-12T12:11:15.740 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 16389, 2021-05-12T12:11:15.740 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T12:11:15.740 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 3, 2021-05-12T12:11:15.740 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:11:15.740 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:11:15.741 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 1, 2021-05-12T12:11:15.741 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:11:15.741 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:11:15.741 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:11:15.742 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:11:15.742 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:11:15.742 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:11:15.742 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 1, 2021-05-12T12:11:15.742 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:11:15.743 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 1, 2021-05-12T12:11:15.743 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 1, 2021-05-12T12:11:15.743 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 1, 2021-05-12T12:11:15.743 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:11:15.744 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:11:15.744 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:11:15.744 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:11:15.744 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:11:15.744 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:11:15.745 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:11:15.745 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.745 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "29", 2021-05-12T12:11:15.745 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:11:15.745 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:11:15.746 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:11:15.746 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:11:15.746 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "pool", 2021-05-12T12:11:15.746 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 1, 2021-05-12T12:11:15.747 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 29, 2021-05-12T12:11:15.747 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [ 2021-05-12T12:11:15.747 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.747 INFO:tasks.workunit.client.0.smithi090.stdout: "snapid": 1, 2021-05-12T12:11:15.747 INFO:tasks.workunit.client.0.smithi090.stdout: "stamp": "2021-05-12T12:10:10.291782+0000", 2021-05-12T12:11:15.748 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "snap1" 2021-05-12T12:11:15.748 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.748 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.748 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:11:15.749 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:11:15.749 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:11:15.749 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:11:15.749 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:11:15.749 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:11:15.750 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:11:15.750 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:11:15.750 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:11:15.750 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:11:15.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:11:15.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:11:15.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:11:15.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:11:15.751 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:11:15.752 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "myprofile", 2021-05-12T12:11:15.752 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:11:15.752 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:11:15.752 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.753 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:11:15.753 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:11:15.753 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:11:15.753 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:11:15.753 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:11:15.754 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:11:15.754 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:11:15.754 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:11:15.754 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 8192, 2021-05-12T12:11:15.754 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:11:15.755 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:11:15.755 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:11:15.756 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:11:15.756 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.756 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.756 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:11:15.757 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.757 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:15.757 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "01e065f1-a569-44af-90fd-e9233051a6cf", 2021-05-12T12:11:15.758 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:11:15.758 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:15.759 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:15.759 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:15.759 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 5, 2021-05-12T12:11:15.759 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:11:15.760 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 36, 2021-05-12T12:11:15.760 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 41, 2021-05-12T12:11:15.760 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 30, 2021-05-12T12:11:15.760 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:15.760 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:15.761 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.761 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.761 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.762 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:11:15.762 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.762 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.762 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.763 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.763 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:11:15.763 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.763 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.764 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.764 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.764 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:15.765 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.765 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.766 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.766 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:11:15.766 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.767 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.767 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.767 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.767 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:11:15.768 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.768 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.768 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.768 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.768 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:15.769 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.769 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.769 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.769 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:11:15.769 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.770 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.770 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.770 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.770 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:11:15.771 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.771 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.771 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.771 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.771 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:15.772 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.772 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.772 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.772 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:11:15.772 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.773 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.773 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.773 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.773 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:11:15.774 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 76874 2021-05-12T12:11:15.774 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.774 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.774 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.774 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/76874", 2021-05-12T12:11:15.775 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/76874", 2021-05-12T12:11:15.775 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/76874", 2021-05-12T12:11:15.775 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/76874", 2021-05-12T12:11:15.775 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:15.775 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:11:15.776 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:11:15.776 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.776 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.776 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.776 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:15.777 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "219c9c7a-4f27-4908-a6a5-bc27283f6f3a", 2021-05-12T12:11:15.777 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:11:15.777 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:15.777 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:15.777 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:15.778 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:11:15.778 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:11:15.778 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 9, 2021-05-12T12:11:15.778 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 24, 2021-05-12T12:11:15.779 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 31, 2021-05-12T12:11:15.779 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:15.779 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:15.779 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.780 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.780 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.780 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:15.780 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.780 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.781 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.781 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.781 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:15.781 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.782 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.782 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.782 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.782 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:15.782 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.783 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.783 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.783 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:15.783 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.784 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.784 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.784 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.784 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:15.784 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.785 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.785 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.785 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.785 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:15.785 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.786 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.786 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.786 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:15.786 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.787 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.787 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.787 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.787 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:15.787 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.788 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.788 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.789 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.789 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:15.789 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.790 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.790 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.790 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:15.790 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.791 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.791 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.791 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.792 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:15.792 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:15.792 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.793 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.793 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.793 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/71068", 2021-05-12T12:11:15.793 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/71068", 2021-05-12T12:11:15.793 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/71068", 2021-05-12T12:11:15.794 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/71068", 2021-05-12T12:11:15.794 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:15.794 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:11:15.794 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.795 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.795 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.795 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:15.795 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "79c730d8-a2c9-4bb0-a0cd-79bdc776a37a", 2021-05-12T12:11:15.795 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:11:15.796 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:15.796 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:15.796 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:15.796 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 14, 2021-05-12T12:11:15.796 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 28, 2021-05-12T12:11:15.797 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 38, 2021-05-12T12:11:15.797 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:11:15.797 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 32, 2021-05-12T12:11:15.797 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:15.798 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:15.798 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.798 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.798 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.799 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:15.799 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.799 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.799 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.799 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.800 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:15.800 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.800 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.800 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.800 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.801 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:15.801 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.801 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.801 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.802 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:15.802 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.802 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.802 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.803 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.803 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:15.803 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.803 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.803 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.804 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.804 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:15.804 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.804 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.804 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.805 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:15.805 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.805 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.805 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.806 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.806 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:15.806 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.806 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.807 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.807 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.807 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:15.807 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:15.807 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.808 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:15.808 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:15.808 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.808 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.808 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.809 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:15.809 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:15.809 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 77457 2021-05-12T12:11:15.809 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.809 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.810 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.810 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/77457", 2021-05-12T12:11:15.810 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/77457", 2021-05-12T12:11:15.810 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/77457", 2021-05-12T12:11:15.811 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/77457", 2021-05-12T12:11:15.811 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:15.811 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:11:15.811 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:11:15.812 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.812 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.812 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.812 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:11:15.812 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.813 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:15.813 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:10:21.266625+0000", 2021-05-12T12:11:15.813 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:15.813 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:15.813 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:15.814 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:15.814 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:05.952240+0000", 2021-05-12T12:11:15.814 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:15.814 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.815 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.815 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:15.815 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:10:21.801656+0000", 2021-05-12T12:11:15.815 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:15.815 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:15.816 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:15.816 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:15.816 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:15.119283+0000", 2021-05-12T12:11:15.816 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:15.817 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.817 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.817 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:15.817 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:10:22.376622+0000", 2021-05-12T12:11:15.817 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:15.818 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:15.818 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:15.818 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:15.818 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:24.675627+0000", 2021-05-12T12:11:15.819 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:15.819 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.819 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.819 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:11:15.819 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:11:15.820 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:11:15.820 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:11:15.820 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:11:15.820 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:11:15.820 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:11:15.821 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:11:15.821 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:11:15.821 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:11:15.821 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:11:15.822 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.822 INFO:tasks.workunit.client.0.smithi090.stdout: "myprofile": { 2021-05-12T12:11:15.822 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-device-class": "", 2021-05-12T12:11:15.822 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-failure-domain": "osd", 2021-05-12T12:11:15.822 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-root": "default", 2021-05-12T12:11:15.823 INFO:tasks.workunit.client.0.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T12:11:15.823 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:11:15.823 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "1", 2021-05-12T12:11:15.823 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:11:15.823 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van", 2021-05-12T12:11:15.824 INFO:tasks.workunit.client.0.smithi090.stdout: "w": "8" 2021-05-12T12:11:15.824 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.824 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.824 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:11:15.824 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:11:15.825 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:11:15.825 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:11:15.825 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:11:15.825 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:11:15.826 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:11:15.826 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:11:15.826 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:11:15.826 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:11:15.826 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:11:15.827 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.827 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.827 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:11:15.827 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.828 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:15.828 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:15.828 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/76874,v1:127.0.0.1:6805/76874]", 2021-05-12T12:11:15.828 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:15.828 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:15.829 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:15.829 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:15.829 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:15.829 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:15.830 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:15.830 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:15.830 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:15.830 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:11:15.831 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:15.831 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:15.831 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:15.831 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:15.831 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:15.832 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:15.832 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:15.832 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:15.832 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:15.833 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:15.833 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:15.833 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:15.833 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:15.833 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:15.834 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:15.834 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/76874,v1:127.0.0.1:6803/76874]", 2021-05-12T12:11:15.834 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:15.834 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/76874,v1:127.0.0.1:6809/76874]", 2021-05-12T12:11:15.835 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/76874,v1:127.0.0.1:6807/76874]", 2021-05-12T12:11:15.835 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:15.835 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:15.835 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:15.835 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:15.836 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:15.836 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:15.836 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:15.837 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:15.837 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:15.837 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:11:15.838 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:15.838 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:15.839 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:15.839 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.840 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.840 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:15.840 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:15.840 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/71068,v1:127.0.0.1:6813/71068]", 2021-05-12T12:11:15.840 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:15.841 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:15.841 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:15.841 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:15.841 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:15.842 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:15.842 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:15.842 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:15.842 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:15.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:11:15.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:15.843 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:15.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:15.844 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:15.844 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:15.844 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:15.844 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:15.845 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:15.845 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:15.845 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:15.845 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:15.845 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:15.846 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:15.846 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:15.846 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:15.846 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/71068,v1:127.0.0.1:6811/71068]", 2021-05-12T12:11:15.847 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:15.847 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/71068,v1:127.0.0.1:6817/71068]", 2021-05-12T12:11:15.847 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/71068,v1:127.0.0.1:6815/71068]", 2021-05-12T12:11:15.847 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:15.847 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:15.848 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:15.848 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:15.848 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:15.848 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:15.848 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:15.849 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:15.849 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:15.849 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:11:15.849 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:15.850 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:15.850 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:15.850 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.850 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.850 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:15.851 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:15.851 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/77457,v1:127.0.0.1:6813/77457]", 2021-05-12T12:11:15.851 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:15.851 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:15.851 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:15.852 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:15.852 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:15.852 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:15.852 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:15.852 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:15.853 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:15.853 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:11:15.853 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:15.853 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:15.854 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:15.854 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:15.854 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:15.854 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:15.854 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:15.855 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:15.855 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:15.855 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:15.855 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:15.855 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:15.856 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:15.856 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:15.856 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:15.856 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/77457,v1:127.0.0.1:6811/77457]", 2021-05-12T12:11:15.857 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:15.857 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/77457,v1:127.0.0.1:6817/77457]", 2021-05-12T12:11:15.857 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/77457,v1:127.0.0.1:6815/77457]", 2021-05-12T12:11:15.858 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:15.858 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:15.858 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:15.858 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:15.859 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:15.859 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:15.859 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:15.859 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:15.859 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:15.860 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:11:15.860 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:15.860 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:15.860 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:15.860 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.861 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.861 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:11:15.861 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:11:15.861 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:11:15.861 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:11:15.862 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.862 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:11:15.862 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.862 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:15.863 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 36 2021-05-12T12:11:15.863 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.863 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.863 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:15.863 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 9 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 38 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.864 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.865 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.865 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:11:15.865 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 43, 2021-05-12T12:11:15.865 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:11:15.866 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:11:15.866 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.866 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:15.866 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:11:15.866 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:15.867 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.867 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.867 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:15.867 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:11:15.867 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:15.868 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.868 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.868 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:15.868 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:11:15.868 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:15.869 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.869 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.869 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:11:15.869 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.869 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:11:15.870 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.871 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.871 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:11:15.871 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:11:15.871 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.871 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.872 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:11:15.872 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:11:15.872 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.872 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.872 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:11:15.873 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:11:15.873 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.873 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.873 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:11:15.873 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:11:15.874 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.874 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.874 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:11:15.874 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:11:15.874 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.875 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.875 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:11:15.875 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:11:15.875 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.875 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.876 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:11:15.876 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:11:15.876 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.876 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:11:15.877 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:11:15.877 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.877 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.877 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:11:15.877 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:11:15.878 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.878 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.878 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:15.878 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:11:15.878 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.879 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.879 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:11:15.879 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.879 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:11:15.879 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:11:15.880 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:15.880 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:11:15.880 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.880 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:15.880 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:15.881 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:15.881 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.881 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:11:15.881 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.881 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:15.882 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.882 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.882 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.882 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.882 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:11:15.883 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:11:15.883 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:15.883 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:11:15.883 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.883 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.884 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:15.885 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.885 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.885 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.885 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.885 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:11:15.886 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:11:15.886 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:15.886 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:11:15.886 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.886 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:15.887 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:15.887 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:15.887 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.887 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:15.887 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.888 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:15.888 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.888 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.888 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:15.888 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.889 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:11:15.889 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.889 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.889 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:15.890 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.890 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:11:15.890 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.890 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.890 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.891 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.891 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:11:15.891 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:11:15.891 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:15.892 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:11:15.892 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:15.892 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:15.892 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:15.892 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:15.893 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.893 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:15.893 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.893 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:15.893 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.894 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.894 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:15.894 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.894 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:11:15.895 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.895 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.895 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:15.895 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:15.895 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:11:15.896 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.896 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.896 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.896 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.896 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:11:15.897 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.897 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:11:15.897 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:11:15.897 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:11:15.897 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:11:15.898 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:11:15.898 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:11:15.898 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:11:15.898 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.898 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:11:15.899 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:11:15.899 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:11:15.899 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.899 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.899 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:11:15.900 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:11:15.900 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:11:15.900 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.900 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.900 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:11:15.901 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.901 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.901 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.901 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.901 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 1, 2021-05-12T12:11:15.902 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "test", 2021-05-12T12:11:15.902 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 1, 2021-05-12T12:11:15.902 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 3, 2021-05-12T12:11:15.902 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 3, 2021-05-12T12:11:15.903 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 3, 2021-05-12T12:11:15.903 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:11:15.903 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.903 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "set_chooseleaf_tries", 2021-05-12T12:11:15.903 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 5 2021-05-12T12:11:15.904 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.904 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.904 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "set_choose_tries", 2021-05-12T12:11:15.904 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 100 2021-05-12T12:11:15.904 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.905 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.905 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:11:15.905 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:11:15.905 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:11:15.905 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.906 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.906 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_indep", 2021-05-12T12:11:15.906 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:11:15.906 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:11:15.907 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.907 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.907 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:11:15.907 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.907 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:15.908 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.908 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.908 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:11:15.908 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:11:15.908 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:11:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:11:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:11:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:11:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:11:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:11:15.910 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:11:15.910 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:11:15.910 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:11:15.910 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:11:15.910 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:11:15.911 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:11:15.911 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:11:15.911 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 1, 2021-05-12T12:11:15.911 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:11:15.911 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:11:15.912 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:11:15.912 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:11:15.912 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:11:15.912 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.912 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:11:15.913 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.913 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:11:15.913 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:11:15.913 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:11:15.913 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:11:15.914 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:11:15.914 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:11:15.914 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:11:15.914 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:11:15.914 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:11:15.915 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:11:15.915 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:11:15.915 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:11:15.915 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:11:15.915 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:11:15.916 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:11:15.916 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:11:15.916 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.916 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.916 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:11:15.917 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:11:15.917 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:11:15.917 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.917 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:11:15.917 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:11:15.918 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.918 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:11:15.918 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:11:15.918 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:11:15.919 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:11:15.919 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:11:15.919 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:11:15.919 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.919 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:11:15.920 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 1, 2021-05-12T12:11:15.920 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:11:15.920 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:11:15.920 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:11:15.920 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:11:15.921 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48430, 2021-05-12T12:11:15.921 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 101, 2021-05-12T12:11:15.921 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 1, 2021-05-12T12:11:15.921 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 303, 2021-05-12T12:11:15.921 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:15.922 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:11:15.922 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:15.922 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:15.922 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:15.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:15.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:11:15.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:11:15.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:11:15.923 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:11:15.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:11:15.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:15.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:15.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:15.924 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:15.925 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:11:15.926 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:11:15.926 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:11:15.926 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:11:15.926 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:11:15.927 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:15.927 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:15.927 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:15.927 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:15.927 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:15.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:15.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:15.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:15.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:15.928 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:11:15.929 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:11:15.929 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.929 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:11:15.929 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:11:15.929 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:11:15.930 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:15.930 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:11:15.930 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:11:15.930 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:15.931 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:15.931 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:15.931 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:11:15.931 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:11:15.931 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.932 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 18, 2021-05-12T12:11:15.932 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:15.932 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 3, 2021-05-12T12:11:15.932 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 3, 2021-05-12T12:11:15.932 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:11:15.933 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.933 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:11:15.933 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:11:15.933 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:11:15.933 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:11:15.934 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:11:15.934 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:15.934 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:15.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:11:15.934 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18636, 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2156, 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16445, 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554164, 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:11:15.935 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:11:15.936 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103463936, 2021-05-12T12:11:15.936 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:15.936 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2207744, 2021-05-12T12:11:15.936 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1546875, 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16840592 2021-05-12T12:11:15.937 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.938 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:11:15.938 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:15.938 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:15.938 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:15.939 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:11:15.939 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:11:15.939 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:11:15.939 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.939 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:11:15.940 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:15.940 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:15.940 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:15.940 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:11:15.940 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.941 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:11:15.941 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:11:15.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.941 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:11:15.941 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:15.942 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:15.943 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:11:15.943 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 18636, 2021-05-12T12:11:15.943 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2156, 2021-05-12T12:11:15.943 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 2, 2021-05-12T12:11:15.943 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 16445, 2021-05-12T12:11:15.944 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314554164, 2021-05-12T12:11:15.944 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:11:15.944 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:11:15.944 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322103463936, 2021-05-12T12:11:15.944 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:15.945 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2207744, 2021-05-12T12:11:15.945 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1546875, 2021-05-12T12:11:15.945 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:15.945 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:15.945 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:15.946 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 2160, 2021-05-12T12:11:15.946 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 16840592 2021-05-12T12:11:15.946 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.946 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:11:15.946 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:15.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:15.947 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:15.947 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:11:15.947 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:11:15.947 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:11:15.948 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.948 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:11:15.948 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:15.948 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:15.948 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:15.949 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:11:15.949 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.949 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:11:15.949 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:11:15.950 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.950 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.950 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:11:15.950 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.950 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:11:15.951 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:11:15.951 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:11:15.951 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48430, 2021-05-12T12:11:15.951 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 101, 2021-05-12T12:11:15.951 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 1, 2021-05-12T12:11:15.952 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 303, 2021-05-12T12:11:15.952 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:15.952 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:11:15.952 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:15.953 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:15.953 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:15.953 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:15.953 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:11:15.953 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:11:15.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:11:15.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:11:15.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:11:15.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:15.954 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:15.955 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:15.955 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:15.955 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:15.955 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:15.955 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:11:15.956 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:15.956 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:15.956 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:11:15.956 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:11:15.956 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:11:15.957 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:11:15.957 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:11:15.957 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:15.957 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:15.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:15.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:15.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:15.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:15.958 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:15.959 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:15.959 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:15.959 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:11:15.959 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:11:15.959 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.960 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:11:15.960 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:11:15.960 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:11:15.960 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:15.960 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1241088, 2021-05-12T12:11:15.961 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1241088, 2021-05-12T12:11:15.961 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:15.961 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:15.961 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:15.962 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:11:15.962 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:11:15.962 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.962 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 18, 2021-05-12T12:11:15.963 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:15.963 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 3, 2021-05-12T12:11:15.963 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 3, 2021-05-12T12:11:15.963 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:11:15.963 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.964 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.964 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:11:15.964 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.964 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:15.964 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 154618822668 2021-05-12T12:11:15.965 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.965 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.965 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:15.965 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 38654705687 2021-05-12T12:11:15.965 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.966 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.966 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:15.966 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 163208757259 2021-05-12T12:11:15.966 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.966 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:11:15.967 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.967 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:11:15.967 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 1 2021-05-12T12:11:15.967 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.968 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:11:15.968 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.968 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:15.968 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:15.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:15.969 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:15.969 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.969 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.969 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2147483647, 2021-05-12T12:11:15.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:15.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:15.970 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:15.970 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.970 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.971 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:15.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:15.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:11:15.971 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:15.971 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.972 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.972 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:15.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 1, 2021-05-12T12:11:15.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:15.972 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:15.973 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.973 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.973 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:11:15.973 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:15.974 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:11:15.974 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:11:15.974 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.974 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:15.974 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:11:15.975 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 5, 2021-05-12T12:11:15.975 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:10:25.537855+0000", 2021-05-12T12:11:15.975 INFO:tasks.workunit.client.0.smithi090.stdout: "services": {} 2021-05-12T12:11:15.975 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:15.975 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:11:15.976 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:11:15.976 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 416, 2021-05-12T12:11:15.976 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:11:15.976 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:11:15.976 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:15.977 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:11:15.979 INFO:tasks.workunit.client.0.smithi090.stderr:report 731637990 2021-05-12T12:11:15.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:11:15.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:15.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2021-05-12T12:11:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:11:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:11:15.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:11:15.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:11:15.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:11:15.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:11:15.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2021-05-12T12:11:15.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:11:15.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:11:15.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=0 2021-05-12T12:11:15.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:11:15.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2021-05-12T12:11:15.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2021-05-12T12:11:16.670 INFO:tasks.workunit.client.0.smithi090.stderr:PG '1.0' not found 2021-05-12T12:11:17.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:17.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T12:11:17.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:17.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:17.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:11:17.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:17.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T12:11:17.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:17.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:17.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:17.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:17.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T12:11:17.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T12:11:17.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:17.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:17.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:17.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:17.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:17.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:17.195 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:17.195 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:17.196 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:11:17.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:17.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:17.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:17.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:17.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:17.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:17.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:17.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:17.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:17.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:17.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T12:11:17.198 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:11:17.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:11:17.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:17.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T12:11:17.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:11:17.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:17.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:17.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:17.221 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:17.218+0000 7faaa71f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:17.223 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:17.221+0000 7faaa71f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:17.224 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:17.222+0000 7faaa71f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:17.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:11:17.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:17.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:11:17.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:17.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:17.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:17.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:17.572 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:11:17.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:17.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:11:17.740 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:17.738+0000 7faaa71f80c0 -1 Falling back to public interface 2021-05-12T12:11:17.955 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/76874,v1:127.0.0.1:6803/76874] [v2:127.0.0.1:6804/76874,v1:127.0.0.1:6805/76874] exists,up 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:11:17.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:17.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:17.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:17.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:17.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T12:11:17.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:11:17.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:11:17.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:11:17.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:11:17.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:11:18.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:11:18.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2021-05-12T12:11:18.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:11:18.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:11:18.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T12:11:18.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:11:18.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T12:11:18.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2021-05-12T12:11:18.903 INFO:tasks.workunit.client.0.smithi090.stderr:PG '1.0' not found 2021-05-12T12:11:19.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:19.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T12:11:19.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:19.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:19.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:11:19.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:19.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T12:11:19.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:19.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:19.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:19.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:19.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T12:11:19.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T12:11:19.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:19.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:19.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:19.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:19.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:19.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:19.428 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:19.428 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:19.428 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:11:19.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:19.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:19.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:19.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:19.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:19.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:19.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:19.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:19.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:19.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:19.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T12:11:19.431 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:11:19.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:11:19.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:19.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T12:11:19.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:11:19.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:19.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:19.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:19.453 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:19.451+0000 7fabd9d5f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:19.455 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:19.453+0000 7fabd9d5f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:19.457 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:19.454+0000 7fabd9d5f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:19.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:11:19.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:11:19.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:11:19.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:11:19.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:11:19.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:11:19.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:11:19.802 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:11:19.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:11:19.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:11:19.973 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:19.971+0000 7fabd9d5f0c0 -1 Falling back to public interface 2021-05-12T12:11:20.004 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:20.002+0000 7faaa71f80c0 -1 osd.0 43 log_to_monitors {default=true} 2021-05-12T12:11:20.175 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [14,28) [v2:127.0.0.1:6810/77457,v1:127.0.0.1:6811/77457] [v2:127.0.0.1:6812/77457,v1:127.0.0.1:6813/77457] exists,up 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:11:20.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:11:20.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:11:20.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:11:20.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2021-05-12T12:11:20.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T12:11:20.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T12:11:20.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T12:11:20.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T12:11:20.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T12:11:20.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2021-05-12T12:11:21.520 INFO:tasks.workunit.client.0.smithi090.stderr:PG '1.0' not found 2021-05-12T12:11:21.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T12:11:21.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T12:11:21.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2021-05-12T12:11:21.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:11:21.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:11:21.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:11:21.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:11:21.792 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T12:11:21.793 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:11:21.794 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:11:21.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:11:21.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:11:21.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:11:21.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:11:21.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:11:21.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:11:21.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:11:21.858 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:11:22.259 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:22.257+0000 7fabd9d5f0c0 -1 osd.2 43 log_to_monitors {default=true} 2021-05-12T12:11:22.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:11:22.271 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:11:22.271 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:11:22.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:11:22.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:22.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:11:22.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=197568495619 2021-05-12T12:11:22.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 197568495619 2021-05-12T12:11:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-197568495619' 2021-05-12T12:11:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:22.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:11:22.577 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-05-12T12:11:22.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:11:22.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:11:22.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:11:22.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:11:22.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:11:22.678 INFO:tasks.workunit.client.0.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T12:11:22.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T12:11:22.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T12:11:22.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T12:11:22.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:11:22.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-197568495619 2021-05-12T12:11:22.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:11:22.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:11:22.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-197568495619 2021-05-12T12:11:22.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:11:22.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=197568495619 2021-05-12T12:11:22.683 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 197568495619 2021-05-12T12:11:22.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2021-05-12T12:11:22.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:11:23.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 197568495619 2021-05-12T12:11:23.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:11:24.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:11:24.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:11:24.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 197568495619 -lt 197568495619 2021-05-12T12:11:24.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:11:24.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:24.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:24.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:11:24.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:24.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:24.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:24.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:24.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:24.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:24.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:25.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:25.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:25.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:25.322 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:25.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:25.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:11:25.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:11:25.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:11:25.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:25.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:11:25.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:25.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:25.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:25.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:25.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:25.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:25.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:25.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:26.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:26.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:26.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:26.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:26.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:26.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:26.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:26.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:27.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:27.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:27.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T12:11:27.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:27.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:11:27.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:27.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:27.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:27.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:27.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:27.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:27.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:27.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:27.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:27.766 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:27.767 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:27.767 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:28.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:28.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:28.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:28.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:28.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:28.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:28.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:28.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:28.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:28.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:28.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:28.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T12:11:28.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:28.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T12:11:29.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:29.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:29.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:29.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:29.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:29.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:29.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:29.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:29.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:29.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:29.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:29.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:29.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:29.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:29.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:29.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:29.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:29.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:29.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:30.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:30.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:30.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T12:11:30.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:30.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T12:11:31.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:31.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:31.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:31.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:31.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:31.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:31.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:31.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:31.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:31.590 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:31.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:31.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:32.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:32.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:32.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:32.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:32.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:32.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:32.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:32.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:32.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:32.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:32.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:32.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T12:11:32.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:32.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T12:11:34.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:34.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:34.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:34.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:34.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:34.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:34.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:34.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:34.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:34.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:34.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:34.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:34.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:34.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:34.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:34.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:34.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:34.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:34.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:34.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:34.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:35.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:35.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:35.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T12:11:35.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:35.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T12:11:38.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:38.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:38.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:38.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:38.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:38.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:38.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:38.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:39.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:39.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:39.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:39.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:39.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:39.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:39.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:39.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:39.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:39.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:39.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:39.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:39.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:40.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:40.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:40.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T12:11:40.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:40.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T12:11:46.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:46.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:46.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:46.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:46.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:46.410 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:46.410 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:46.410 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:46.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:46.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:46.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:46.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:47.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:47.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:47.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:47.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:47.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:47.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:47.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:47.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:47.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:47.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:47.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:47.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T12:11:47.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:11:47.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T12:11:55.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:11:55.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:11:55.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:11:55.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:11:55.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:11:55.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:11:55.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:11:55.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:11:55.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:11:55.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:11:55.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:11:55.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:11:55.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 1 2021-05-12T12:11:55.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:11:55.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:11:55.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:11:55.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:11:55.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:11:55.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:11:55.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:11:55.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:11:56.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:11:56.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:11:56.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T12:11:56.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T12:11:56.839 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:11:56.839 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_fingerprint": "05499b63-48ca-400c-9b23-2220766a714e", 2021-05-12T12:11:56.839 INFO:tasks.workunit.client.0.smithi090.stdout: "version": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:56.840 INFO:tasks.workunit.client.0.smithi090.stdout: "commit": "36c2cf2fb22a9790b2c097e158985773b35f1740", 2021-05-12T12:11:56.840 INFO:tasks.workunit.client.0.smithi090.stdout: "timestamp": "2021-05-12T12:11:56.835494+0000", 2021-05-12T12:11:56.840 INFO:tasks.workunit.client.0.smithi090.stdout: "tag": "", 2021-05-12T12:11:56.840 INFO:tasks.workunit.client.0.smithi090.stdout: "health": { 2021-05-12T12:11:56.840 INFO:tasks.workunit.client.0.smithi090.stdout: "status": "HEALTH_ERR", 2021-05-12T12:11:56.841 INFO:tasks.workunit.client.0.smithi090.stdout: "checks": { 2021-05-12T12:11:56.841 INFO:tasks.workunit.client.0.smithi090.stdout: "MGR_MODULE_ERROR": { 2021-05-12T12:11:56.841 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_ERR", 2021-05-12T12:11:56.841 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:56.842 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'", 2021-05-12T12:11:56.842 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:56.842 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.842 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:56.843 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.843 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Module 'devicehealth' has failed: 'option not found: mgr_pool'" 2021-05-12T12:11:56.843 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.843 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.844 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:56.844 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.844 INFO:tasks.workunit.client.0.smithi090.stdout: "OSDMAP_FLAGS": { 2021-05-12T12:11:56.844 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:56.845 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:56.845 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T12:11:56.845 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 17 2021-05-12T12:11:56.845 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.845 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [], 2021-05-12T12:11:56.846 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:56.846 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.846 INFO:tasks.workunit.client.0.smithi090.stdout: "OSD_DOWN": { 2021-05-12T12:11:56.846 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:56.846 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:56.847 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 osds down", 2021-05-12T12:11:56.847 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:56.847 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.847 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:56.847 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.848 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "osd.1 (root=default,host=smithi090) is down" 2021-05-12T12:11:56.848 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.848 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.848 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:56.848 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.849 INFO:tasks.workunit.client.0.smithi090.stdout: "PG_DEGRADED": { 2021-05-12T12:11:56.849 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:56.849 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:56.849 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-05-12T12:11:56.850 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:56.850 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.850 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:56.850 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.851 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-05-12T12:11:56.851 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.852 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.852 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:56.853 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.853 INFO:tasks.workunit.client.0.smithi090.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T12:11:56.853 INFO:tasks.workunit.client.0.smithi090.stdout: "severity": "HEALTH_WARN", 2021-05-12T12:11:56.854 INFO:tasks.workunit.client.0.smithi090.stdout: "summary": { 2021-05-12T12:11:56.854 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T12:11:56.855 INFO:tasks.workunit.client.0.smithi090.stdout: "count": 1 2021-05-12T12:11:56.855 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.855 INFO:tasks.workunit.client.0.smithi090.stdout: "detail": [ 2021-05-12T12:11:56.855 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.855 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T12:11:56.856 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.856 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.856 INFO:tasks.workunit.client.0.smithi090.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T12:11:56.856 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.857 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.857 INFO:tasks.workunit.client.0.smithi090.stdout: "muted": false 2021-05-12T12:11:56.857 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.857 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.857 INFO:tasks.workunit.client.0.smithi090.stdout: "mutes": [] 2021-05-12T12:11:56.858 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.858 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_first_committed": 1, 2021-05-12T12:11:56.858 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap_last_committed": 1, 2021-05-12T12:11:56.858 INFO:tasks.workunit.client.0.smithi090.stdout: "monmap": { 2021-05-12T12:11:56.859 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:11:56.859 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:11:56.859 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:08:55.755262Z", 2021-05-12T12:11:56.859 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:08:55.755262Z", 2021-05-12T12:11:56.859 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release": 17, 2021-05-12T12:11:56.860 INFO:tasks.workunit.client.0.smithi090.stdout: "min_mon_release_name": "quincy", 2021-05-12T12:11:56.860 INFO:tasks.workunit.client.0.smithi090.stdout: "election_strategy": 1, 2021-05-12T12:11:56.860 INFO:tasks.workunit.client.0.smithi090.stdout: "disallowed_leaders: ": "", 2021-05-12T12:11:56.861 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": false, 2021-05-12T12:11:56.861 INFO:tasks.workunit.client.0.smithi090.stdout: "features": { 2021-05-12T12:11:56.861 INFO:tasks.workunit.client.0.smithi090.stdout: "persistent": [ 2021-05-12T12:11:56.862 INFO:tasks.workunit.client.0.smithi090.stdout: "kraken", 2021-05-12T12:11:56.862 INFO:tasks.workunit.client.0.smithi090.stdout: "luminous", 2021-05-12T12:11:56.862 INFO:tasks.workunit.client.0.smithi090.stdout: "mimic", 2021-05-12T12:11:56.862 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap-prune", 2021-05-12T12:11:56.863 INFO:tasks.workunit.client.0.smithi090.stdout: "nautilus", 2021-05-12T12:11:56.863 INFO:tasks.workunit.client.0.smithi090.stdout: "octopus", 2021-05-12T12:11:56.863 INFO:tasks.workunit.client.0.smithi090.stdout: "pacific", 2021-05-12T12:11:56.863 INFO:tasks.workunit.client.0.smithi090.stdout: "elector-pinging", 2021-05-12T12:11:56.863 INFO:tasks.workunit.client.0.smithi090.stdout: "quincy" 2021-05-12T12:11:56.864 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.864 INFO:tasks.workunit.client.0.smithi090.stdout: "optional": [] 2021-05-12T12:11:56.864 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.864 INFO:tasks.workunit.client.0.smithi090.stdout: "mons": [ 2021-05-12T12:11:56.865 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.865 INFO:tasks.workunit.client.0.smithi090.stdout: "rank": 0, 2021-05-12T12:11:56.865 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "a", 2021-05-12T12:11:56.865 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:56.865 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.866 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.866 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.866 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115", 2021-05-12T12:11:56.866 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 0 2021-05-12T12:11:56.867 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.867 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.867 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.867 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T12:11:56.867 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T12:11:56.868 INFO:tasks.workunit.client.0.smithi090.stdout: "priority": 0, 2021-05-12T12:11:56.868 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 0, 2021-05-12T12:11:56.868 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_location": "{}" 2021-05-12T12:11:56.868 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.868 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.869 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.869 INFO:tasks.workunit.client.0.smithi090.stdout: "quorum": [ 2021-05-12T12:11:56.869 INFO:tasks.workunit.client.0.smithi090.stdout: 0 2021-05-12T12:11:56.869 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.870 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap": { 2021-05-12T12:11:56.870 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 51, 2021-05-12T12:11:56.870 INFO:tasks.workunit.client.0.smithi090.stdout: "fsid": "8160268e-296f-4dfd-917e-d0d3cee77a07", 2021-05-12T12:11:56.870 INFO:tasks.workunit.client.0.smithi090.stdout: "created": "2021-05-12T12:08:56.622276+0000", 2021-05-12T12:11:56.870 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:11:22.859491+0000", 2021-05-12T12:11:56.871 INFO:tasks.workunit.client.0.smithi090.stdout: "last_up_change": "2021-05-12T12:11:22.614179+0000", 2021-05-12T12:11:56.871 INFO:tasks.workunit.client.0.smithi090.stdout: "last_in_change": "2021-05-12T12:10:34.503488+0000", 2021-05-12T12:11:56.871 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T12:11:56.871 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_num": 5800384, 2021-05-12T12:11:56.872 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_set": [ 2021-05-12T12:11:56.872 INFO:tasks.workunit.client.0.smithi090.stdout: "nodown", 2021-05-12T12:11:56.872 INFO:tasks.workunit.client.0.smithi090.stdout: "noin", 2021-05-12T12:11:56.872 INFO:tasks.workunit.client.0.smithi090.stdout: "noout", 2021-05-12T12:11:56.873 INFO:tasks.workunit.client.0.smithi090.stdout: "pglog_hardlimit", 2021-05-12T12:11:56.873 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snapdirs", 2021-05-12T12:11:56.873 INFO:tasks.workunit.client.0.smithi090.stdout: "recovery_deletes", 2021-05-12T12:11:56.873 INFO:tasks.workunit.client.0.smithi090.stdout: "sortbitwise" 2021-05-12T12:11:56.873 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.874 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_version": 12, 2021-05-12T12:11:56.874 INFO:tasks.workunit.client.0.smithi090.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T12:11:56.874 INFO:tasks.workunit.client.0.smithi090.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T12:11:56.874 INFO:tasks.workunit.client.0.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T12:11:56.875 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_snapshot": "", 2021-05-12T12:11:56.875 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_max": 1, 2021-05-12T12:11:56.875 INFO:tasks.workunit.client.0.smithi090.stdout: "max_osd": 3, 2021-05-12T12:11:56.875 INFO:tasks.workunit.client.0.smithi090.stdout: "require_min_compat_client": "luminous", 2021-05-12T12:11:56.875 INFO:tasks.workunit.client.0.smithi090.stdout: "min_compat_client": "jewel", 2021-05-12T12:11:56.876 INFO:tasks.workunit.client.0.smithi090.stdout: "require_osd_release": "quincy", 2021-05-12T12:11:56.876 INFO:tasks.workunit.client.0.smithi090.stdout: "pools": [ 2021-05-12T12:11:56.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.876 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:11:56.877 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_name": "test", 2021-05-12T12:11:56.877 INFO:tasks.workunit.client.0.smithi090.stdout: "create_time": "2021-05-12T12:09:27.835639+0000", 2021-05-12T12:11:56.877 INFO:tasks.workunit.client.0.smithi090.stdout: "flags": 16389, 2021-05-12T12:11:56.877 INFO:tasks.workunit.client.0.smithi090.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T12:11:56.877 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 3, 2021-05-12T12:11:56.878 INFO:tasks.workunit.client.0.smithi090.stdout: "size": 3, 2021-05-12T12:11:56.878 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 2, 2021-05-12T12:11:56.878 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_rule": 1, 2021-05-12T12:11:56.878 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_count": 0, 2021-05-12T12:11:56.878 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_target": 0, 2021-05-12T12:11:56.879 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T12:11:56.879 INFO:tasks.workunit.client.0.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T12:11:56.879 INFO:tasks.workunit.client.0.smithi090.stdout: "object_hash": 2, 2021-05-12T12:11:56.879 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_autoscale_mode": "off", 2021-05-12T12:11:56.879 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num": 1, 2021-05-12T12:11:56.880 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num": 1, 2021-05-12T12:11:56.880 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_placement_num_target": 1, 2021-05-12T12:11:56.880 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_target": 1, 2021-05-12T12:11:56.880 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_num_pending": 1, 2021-05-12T12:11:56.880 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pg_merge_meta": { 2021-05-12T12:11:56.881 INFO:tasks.workunit.client.0.smithi090.stdout: "source_pgid": "0.0", 2021-05-12T12:11:56.881 INFO:tasks.workunit.client.0.smithi090.stdout: "ready_epoch": 0, 2021-05-12T12:11:56.881 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_started": 0, 2021-05-12T12:11:56.881 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": 0, 2021-05-12T12:11:56.881 INFO:tasks.workunit.client.0.smithi090.stdout: "source_version": "0'0", 2021-05-12T12:11:56.882 INFO:tasks.workunit.client.0.smithi090.stdout: "target_version": "0'0" 2021-05-12T12:11:56.882 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.882 INFO:tasks.workunit.client.0.smithi090.stdout: "last_change": "29", 2021-05-12T12:11:56.882 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend": "0", 2021-05-12T12:11:56.882 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T12:11:56.883 INFO:tasks.workunit.client.0.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T12:11:56.883 INFO:tasks.workunit.client.0.smithi090.stdout: "auid": 0, 2021-05-12T12:11:56.883 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_mode": "pool", 2021-05-12T12:11:56.883 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_seq": 1, 2021-05-12T12:11:56.883 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_epoch": 29, 2021-05-12T12:11:56.884 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_snaps": [ 2021-05-12T12:11:56.884 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.884 INFO:tasks.workunit.client.0.smithi090.stdout: "snapid": 1, 2021-05-12T12:11:56.884 INFO:tasks.workunit.client.0.smithi090.stdout: "stamp": "2021-05-12T12:10:10.291782+0000", 2021-05-12T12:11:56.884 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "snap1" 2021-05-12T12:11:56.885 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.885 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.885 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps": "[]", 2021-05-12T12:11:56.885 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_bytes": 0, 2021-05-12T12:11:56.886 INFO:tasks.workunit.client.0.smithi090.stdout: "quota_max_objects": 0, 2021-05-12T12:11:56.886 INFO:tasks.workunit.client.0.smithi090.stdout: "tiers": [], 2021-05-12T12:11:56.886 INFO:tasks.workunit.client.0.smithi090.stdout: "tier_of": -1, 2021-05-12T12:11:56.886 INFO:tasks.workunit.client.0.smithi090.stdout: "read_tier": -1, 2021-05-12T12:11:56.886 INFO:tasks.workunit.client.0.smithi090.stdout: "write_tier": -1, 2021-05-12T12:11:56.887 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_mode": "none", 2021-05-12T12:11:56.887 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_bytes": 0, 2021-05-12T12:11:56.887 INFO:tasks.workunit.client.0.smithi090.stdout: "target_max_objects": 0, 2021-05-12T12:11:56.887 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T12:11:56.888 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T12:11:56.888 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T12:11:56.888 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_flush_age": 0, 2021-05-12T12:11:56.888 INFO:tasks.workunit.client.0.smithi090.stdout: "cache_min_evict_age": 0, 2021-05-12T12:11:56.889 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profile": "myprofile", 2021-05-12T12:11:56.889 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_params": { 2021-05-12T12:11:56.889 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "none" 2021-05-12T12:11:56.889 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.889 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_period": 0, 2021-05-12T12:11:56.890 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_count": 0, 2021-05-12T12:11:56.890 INFO:tasks.workunit.client.0.smithi090.stdout: "use_gmt_hitset": true, 2021-05-12T12:11:56.890 INFO:tasks.workunit.client.0.smithi090.stdout: "min_read_recency_for_promote": 0, 2021-05-12T12:11:56.890 INFO:tasks.workunit.client.0.smithi090.stdout: "min_write_recency_for_promote": 0, 2021-05-12T12:11:56.890 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T12:11:56.891 INFO:tasks.workunit.client.0.smithi090.stdout: "hit_set_search_last_n": 0, 2021-05-12T12:11:56.891 INFO:tasks.workunit.client.0.smithi090.stdout: "grade_table": [], 2021-05-12T12:11:56.891 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_width": 8192, 2021-05-12T12:11:56.891 INFO:tasks.workunit.client.0.smithi090.stdout: "expected_num_objects": 0, 2021-05-12T12:11:56.892 INFO:tasks.workunit.client.0.smithi090.stdout: "fast_read": false, 2021-05-12T12:11:56.892 INFO:tasks.workunit.client.0.smithi090.stdout: "options": {}, 2021-05-12T12:11:56.892 INFO:tasks.workunit.client.0.smithi090.stdout: "application_metadata": {} 2021-05-12T12:11:56.892 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.892 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.893 INFO:tasks.workunit.client.0.smithi090.stdout: "osds": [ 2021-05-12T12:11:56.893 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.893 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:56.893 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "01e065f1-a569-44af-90fd-e9233051a6cf", 2021-05-12T12:11:56.893 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:11:56.894 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:56.894 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:56.894 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:56.894 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 36, 2021-05-12T12:11:56.894 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 43, 2021-05-12T12:11:56.895 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 46, 2021-05-12T12:11:56.895 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 50, 2021-05-12T12:11:56.895 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 45, 2021-05-12T12:11:56.895 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:56.895 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:56.896 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.896 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.896 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.896 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6802", 2021-05-12T12:11:56.896 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.897 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.897 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.897 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.897 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6803", 2021-05-12T12:11:56.897 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.898 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.898 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.898 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.898 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:56.899 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.899 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.899 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.899 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6804", 2021-05-12T12:11:56.899 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.900 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.900 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.900 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.900 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6805", 2021-05-12T12:11:56.901 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.901 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.901 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.901 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.901 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:56.902 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.902 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.902 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.902 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6808", 2021-05-12T12:11:56.903 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.903 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.903 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.903 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.903 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6809", 2021-05-12T12:11:56.904 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.904 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.904 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.904 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.904 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:56.905 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.905 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.905 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.905 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6806", 2021-05-12T12:11:56.905 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.906 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.906 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.906 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.906 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6807", 2021-05-12T12:11:56.906 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 79511 2021-05-12T12:11:56.907 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.907 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.907 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.907 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6803/79511", 2021-05-12T12:11:56.908 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6805/79511", 2021-05-12T12:11:56.908 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6809/79511", 2021-05-12T12:11:56.908 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6807/79511", 2021-05-12T12:11:56.908 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:56.908 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:56.909 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "219c9c7a-4f27-4908-a6a5-bc27283f6f3a", 2021-05-12T12:11:56.910 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 0, 2021-05-12T12:11:56.910 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:56.910 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:56.910 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:56.911 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 0, 2021-05-12T12:11:56.911 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 0, 2021-05-12T12:11:56.911 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 9, 2021-05-12T12:11:56.911 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 24, 2021-05-12T12:11:56.911 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 31, 2021-05-12T12:11:56.912 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:56.912 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:56.912 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.912 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.912 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.913 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:56.913 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.913 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.913 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.913 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.914 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:56.914 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.914 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.914 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.915 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.915 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:56.915 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.915 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.916 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.916 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:56.916 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.916 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.916 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.917 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.917 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:56.917 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.917 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.917 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.918 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.918 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:56.918 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.918 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.918 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.919 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:56.919 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.919 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.919 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.919 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.920 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:56.920 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.920 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.920 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.920 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.921 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:56.921 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.921 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.921 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.921 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:56.922 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.922 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.922 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.922 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.922 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:56.923 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 71068 2021-05-12T12:11:56.923 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.923 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.923 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.923 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/71068", 2021-05-12T12:11:56.924 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/71068", 2021-05-12T12:11:56.924 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/71068", 2021-05-12T12:11:56.924 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/71068", 2021-05-12T12:11:56.924 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:56.924 INFO:tasks.workunit.client.0.smithi090.stdout: "exists" 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: "uuid": "79c730d8-a2c9-4bb0-a0cd-79bdc776a37a", 2021-05-12T12:11:56.925 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 1, 2021-05-12T12:11:56.926 INFO:tasks.workunit.client.0.smithi090.stdout: "in": 1, 2021-05-12T12:11:56.926 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 1, 2021-05-12T12:11:56.926 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_affinity": 1, 2021-05-12T12:11:56.926 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_begin": 38, 2021-05-12T12:11:56.927 INFO:tasks.workunit.client.0.smithi090.stdout: "last_clean_end": 43, 2021-05-12T12:11:56.927 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 50, 2021-05-12T12:11:56.927 INFO:tasks.workunit.client.0.smithi090.stdout: "up_thru": 0, 2021-05-12T12:11:56.927 INFO:tasks.workunit.client.0.smithi090.stdout: "down_at": 49, 2021-05-12T12:11:56.927 INFO:tasks.workunit.client.0.smithi090.stdout: "lost_at": 0, 2021-05-12T12:11:56.928 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addrs": { 2021-05-12T12:11:56.928 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.928 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.928 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.928 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6810", 2021-05-12T12:11:56.929 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.929 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.929 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.929 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.929 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6811", 2021-05-12T12:11:56.930 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.930 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.930 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.930 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.930 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addrs": { 2021-05-12T12:11:56.931 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.931 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.931 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.931 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6812", 2021-05-12T12:11:56.932 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.932 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.932 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.932 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.932 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6813", 2021-05-12T12:11:56.933 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.933 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.933 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.933 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.933 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addrs": { 2021-05-12T12:11:56.934 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.934 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.934 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.934 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6816", 2021-05-12T12:11:56.934 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.935 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.935 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.935 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.935 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6817", 2021-05-12T12:11:56.935 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addrs": { 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: "addrvec": [ 2021-05-12T12:11:56.936 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.937 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v2", 2021-05-12T12:11:56.937 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6814", 2021-05-12T12:11:56.937 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.937 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.937 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.938 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "v1", 2021-05-12T12:11:56.938 INFO:tasks.workunit.client.0.smithi090.stdout: "addr": "127.0.0.1:6815", 2021-05-12T12:11:56.938 INFO:tasks.workunit.client.0.smithi090.stdout: "nonce": 80194 2021-05-12T12:11:56.938 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.938 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.939 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.939 INFO:tasks.workunit.client.0.smithi090.stdout: "public_addr": "127.0.0.1:6811/80194", 2021-05-12T12:11:56.939 INFO:tasks.workunit.client.0.smithi090.stdout: "cluster_addr": "127.0.0.1:6813/80194", 2021-05-12T12:11:56.939 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_back_addr": "127.0.0.1:6817/80194", 2021-05-12T12:11:56.939 INFO:tasks.workunit.client.0.smithi090.stdout: "heartbeat_front_addr": "127.0.0.1:6815/80194", 2021-05-12T12:11:56.940 INFO:tasks.workunit.client.0.smithi090.stdout: "state": [ 2021-05-12T12:11:56.940 INFO:tasks.workunit.client.0.smithi090.stdout: "exists", 2021-05-12T12:11:56.940 INFO:tasks.workunit.client.0.smithi090.stdout: "up" 2021-05-12T12:11:56.940 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.940 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.941 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.941 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_xinfo": [ 2021-05-12T12:11:56.941 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.941 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:56.941 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:11:20.286750+0000", 2021-05-12T12:11:56.942 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:56.942 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:56.942 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:56.942 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:56.943 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:05.952240+0000", 2021-05-12T12:11:56.943 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:56.943 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.943 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.943 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:56.944 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:10:21.801656+0000", 2021-05-12T12:11:56.944 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:56.944 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:56.944 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:56.944 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:56.945 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:15.119283+0000", 2021-05-12T12:11:56.945 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:56.945 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.945 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.945 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:56.946 INFO:tasks.workunit.client.0.smithi090.stdout: "down_stamp": "2021-05-12T12:11:22.439026+0000", 2021-05-12T12:11:56.946 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_probability": 0, 2021-05-12T12:11:56.946 INFO:tasks.workunit.client.0.smithi090.stdout: "laggy_interval": 0, 2021-05-12T12:11:56.946 INFO:tasks.workunit.client.0.smithi090.stdout: "features": 4540138303579357183, 2021-05-12T12:11:56.947 INFO:tasks.workunit.client.0.smithi090.stdout: "old_weight": 0, 2021-05-12T12:11:56.947 INFO:tasks.workunit.client.0.smithi090.stdout: "last_purged_snaps_scrub": "2021-05-12T12:09:24.675627+0000", 2021-05-12T12:11:56.947 INFO:tasks.workunit.client.0.smithi090.stdout: "dead_epoch": 0 2021-05-12T12:11:56.947 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.947 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.948 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap": [], 2021-05-12T12:11:56.948 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_upmap_items": [], 2021-05-12T12:11:56.948 INFO:tasks.workunit.client.0.smithi090.stdout: "pg_temp": [], 2021-05-12T12:11:56.948 INFO:tasks.workunit.client.0.smithi090.stdout: "primary_temp": [], 2021-05-12T12:11:56.948 INFO:tasks.workunit.client.0.smithi090.stdout: "blocklist": {}, 2021-05-12T12:11:56.949 INFO:tasks.workunit.client.0.smithi090.stdout: "erasure_code_profiles": { 2021-05-12T12:11:56.949 INFO:tasks.workunit.client.0.smithi090.stdout: "default": { 2021-05-12T12:11:56.949 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:11:56.949 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "2", 2021-05-12T12:11:56.949 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:11:56.950 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van" 2021-05-12T12:11:56.950 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.950 INFO:tasks.workunit.client.0.smithi090.stdout: "myprofile": { 2021-05-12T12:11:56.950 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-device-class": "", 2021-05-12T12:11:56.950 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-failure-domain": "osd", 2021-05-12T12:11:56.951 INFO:tasks.workunit.client.0.smithi090.stdout: "crush-root": "default", 2021-05-12T12:11:56.951 INFO:tasks.workunit.client.0.smithi090.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T12:11:56.951 INFO:tasks.workunit.client.0.smithi090.stdout: "k": "2", 2021-05-12T12:11:56.951 INFO:tasks.workunit.client.0.smithi090.stdout: "m": "1", 2021-05-12T12:11:56.951 INFO:tasks.workunit.client.0.smithi090.stdout: "plugin": "jerasure", 2021-05-12T12:11:56.952 INFO:tasks.workunit.client.0.smithi090.stdout: "technique": "reed_sol_van", 2021-05-12T12:11:56.952 INFO:tasks.workunit.client.0.smithi090.stdout: "w": "8" 2021-05-12T12:11:56.952 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.952 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.952 INFO:tasks.workunit.client.0.smithi090.stdout: "removed_snaps_queue": [], 2021-05-12T12:11:56.953 INFO:tasks.workunit.client.0.smithi090.stdout: "new_removed_snaps": [], 2021-05-12T12:11:56.953 INFO:tasks.workunit.client.0.smithi090.stdout: "new_purged_snaps": [], 2021-05-12T12:11:56.953 INFO:tasks.workunit.client.0.smithi090.stdout: "crush_node_flags": {}, 2021-05-12T12:11:56.953 INFO:tasks.workunit.client.0.smithi090.stdout: "device_class_flags": {}, 2021-05-12T12:11:56.953 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode": { 2021-05-12T12:11:56.954 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_enabled": false, 2021-05-12T12:11:56.954 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_bucket_count": 0, 2021-05-12T12:11:56.954 INFO:tasks.workunit.client.0.smithi090.stdout: "degraded_stretch_mode": 0, 2021-05-12T12:11:56.954 INFO:tasks.workunit.client.0.smithi090.stdout: "recovering_stretch_mode": 0, 2021-05-12T12:11:56.954 INFO:tasks.workunit.client.0.smithi090.stdout: "stretch_mode_bucket": 0 2021-05-12T12:11:56.955 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.955 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.955 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_metadata": [ 2021-05-12T12:11:56.955 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.955 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:56.956 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:56.956 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6804/79511,v1:127.0.0.1:6805/79511]", 2021-05-12T12:11:56.956 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:56.956 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:56.956 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:56.957 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:56.957 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:56.957 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:56.957 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:56.957 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:56.958 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:56.958 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T12:11:56.958 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:56.958 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:56.959 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:56.959 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:56.959 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:56.959 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:56.960 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:56.960 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:56.960 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:56.960 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:56.960 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:56.961 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:56.961 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:56.961 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:56.962 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:56.962 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6802/79511,v1:127.0.0.1:6803/79511]", 2021-05-12T12:11:56.962 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:56.962 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/79511,v1:127.0.0.1:6809/79511]", 2021-05-12T12:11:56.963 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/79511,v1:127.0.0.1:6807/79511]", 2021-05-12T12:11:56.963 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:56.963 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:56.964 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:56.964 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:56.964 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:56.964 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:56.965 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:56.965 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:56.965 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:56.965 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T12:11:56.965 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:56.966 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:56.966 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:56.966 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.966 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.967 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:56.967 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:56.967 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/71068,v1:127.0.0.1:6813/71068]", 2021-05-12T12:11:56.967 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:56.967 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:56.968 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:56.968 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:56.968 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:56.968 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:56.968 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:56.969 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:56.969 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:56.969 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T12:11:56.969 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:56.970 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:56.970 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:56.970 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:56.970 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:56.970 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:56.971 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:56.971 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:56.971 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:56.971 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:56.971 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/71068,v1:127.0.0.1:6811/71068]", 2021-05-12T12:11:56.972 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:56.973 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/71068,v1:127.0.0.1:6817/71068]", 2021-05-12T12:11:56.973 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/71068,v1:127.0.0.1:6815/71068]", 2021-05-12T12:11:56.973 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:56.973 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:56.974 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:56.974 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:56.974 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:56.974 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:56.974 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:56.975 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:56.975 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:56.975 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T12:11:56.975 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:56.975 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:56.976 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:56.976 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.976 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.976 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:56.976 INFO:tasks.workunit.client.0.smithi090.stdout: "arch": "x86_64", 2021-05-12T12:11:56.977 INFO:tasks.workunit.client.0.smithi090.stdout: "back_addr": "[v2:127.0.0.1:6812/80194,v1:127.0.0.1:6813/80194]", 2021-05-12T12:11:56.977 INFO:tasks.workunit.client.0.smithi090.stdout: "back_iface": "", 2021-05-12T12:11:56.977 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs": "1", 2021-05-12T12:11:56.977 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_db": "0", 2021-05-12T12:11:56.977 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluefs_single_shared_device": "1", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:11:56.978 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T12:11:56.979 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:11:56.979 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:11:56.979 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:11:56.979 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:11:56.979 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_release": "quincy", 2021-05-12T12:11:56.980 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version": "ceph version 17.0.0-4087-g36c2cf2f (36c2cf2fb22a9790b2c097e158985773b35f1740) quincy (dev)", 2021-05-12T12:11:56.980 INFO:tasks.workunit.client.0.smithi090.stdout: "ceph_version_short": "17.0.0-4087-g36c2cf2f", 2021-05-12T12:11:56.980 INFO:tasks.workunit.client.0.smithi090.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-05-12T12:11:56.980 INFO:tasks.workunit.client.0.smithi090.stdout: "default_device_class": "hdd", 2021-05-12T12:11:56.981 INFO:tasks.workunit.client.0.smithi090.stdout: "device_ids": "sda=Hitachi_HUA72201_JPW9J0N20ZVK9C", 2021-05-12T12:11:56.981 INFO:tasks.workunit.client.0.smithi090.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T12:11:56.981 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": "sda", 2021-05-12T12:11:56.981 INFO:tasks.workunit.client.0.smithi090.stdout: "distro": "centos", 2021-05-12T12:11:56.981 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T12:11:56.982 INFO:tasks.workunit.client.0.smithi090.stdout: "distro_version": "8", 2021-05-12T12:11:56.982 INFO:tasks.workunit.client.0.smithi090.stdout: "front_addr": "[v2:127.0.0.1:6810/80194,v1:127.0.0.1:6811/80194]", 2021-05-12T12:11:56.982 INFO:tasks.workunit.client.0.smithi090.stdout: "front_iface": "", 2021-05-12T12:11:56.982 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/80194,v1:127.0.0.1:6817/80194]", 2021-05-12T12:11:56.982 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/80194,v1:127.0.0.1:6815/80194]", 2021-05-12T12:11:56.983 INFO:tasks.workunit.client.0.smithi090.stdout: "hostname": "smithi090", 2021-05-12T12:11:56.983 INFO:tasks.workunit.client.0.smithi090.stdout: "journal_rotational": "1", 2021-05-12T12:11:56.983 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T12:11:56.983 INFO:tasks.workunit.client.0.smithi090.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T12:11:56.983 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_swap_kb": "0", 2021-05-12T12:11:56.984 INFO:tasks.workunit.client.0.smithi090.stdout: "mem_total_kb": "32611924", 2021-05-12T12:11:56.984 INFO:tasks.workunit.client.0.smithi090.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T12:11:56.984 INFO:tasks.workunit.client.0.smithi090.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T12:11:56.984 INFO:tasks.workunit.client.0.smithi090.stdout: "os": "Linux", 2021-05-12T12:11:56.984 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T12:11:56.985 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:11:56.985 INFO:tasks.workunit.client.0.smithi090.stdout: "osdspec_affinity": "", 2021-05-12T12:11:56.985 INFO:tasks.workunit.client.0.smithi090.stdout: "rotational": "1" 2021-05-12T12:11:56.985 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.985 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.986 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_clean_epochs": { 2021-05-12T12:11:56.986 INFO:tasks.workunit.client.0.smithi090.stdout: "min_last_epoch_clean": 0, 2021-05-12T12:11:56.986 INFO:tasks.workunit.client.0.smithi090.stdout: "last_epoch_clean": { 2021-05-12T12:11:56.986 INFO:tasks.workunit.client.0.smithi090.stdout: "per_pool": [] 2021-05-12T12:11:56.986 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.987 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_epochs": [ 2021-05-12T12:11:56.987 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.987 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:56.987 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 47 2021-05-12T12:11:56.987 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.988 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.988 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:56.988 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 9 2021-05-12T12:11:56.988 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.988 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.989 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:56.989 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 51 2021-05-12T12:11:56.989 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.989 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:56.989 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.990 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_first_committed": 1, 2021-05-12T12:11:56.990 INFO:tasks.workunit.client.0.smithi090.stdout: "osdmap_last_committed": 51, 2021-05-12T12:11:56.990 INFO:tasks.workunit.client.0.smithi090.stdout: "crushmap": { 2021-05-12T12:11:56.990 INFO:tasks.workunit.client.0.smithi090.stdout: "devices": [ 2021-05-12T12:11:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.0", 2021-05-12T12:11:56.991 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.1", 2021-05-12T12:11:56.992 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:56.993 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd.2", 2021-05-12T12:11:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: "class": "hdd" 2021-05-12T12:11:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: "types": [ 2021-05-12T12:11:56.994 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 0, 2021-05-12T12:11:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "osd" 2021-05-12T12:11:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.995 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "host" 2021-05-12T12:11:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.996 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 2, 2021-05-12T12:11:56.997 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "chassis" 2021-05-12T12:11:56.997 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.997 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.997 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 3, 2021-05-12T12:11:56.997 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "rack" 2021-05-12T12:11:56.998 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.998 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.998 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 4, 2021-05-12T12:11:56.998 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "row" 2021-05-12T12:11:56.998 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:56.999 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:56.999 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 5, 2021-05-12T12:11:56.999 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pdu" 2021-05-12T12:11:56.999 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 6, 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "pod" 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.000 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 7, 2021-05-12T12:11:57.001 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "room" 2021-05-12T12:11:57.001 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.001 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.001 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 8, 2021-05-12T12:11:57.001 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "datacenter" 2021-05-12T12:11:57.002 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.002 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.002 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 9, 2021-05-12T12:11:57.002 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "zone" 2021-05-12T12:11:57.002 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.003 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.003 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 10, 2021-05-12T12:11:57.003 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "region" 2021-05-12T12:11:57.003 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.003 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.004 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:57.004 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "root" 2021-05-12T12:11:57.004 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.004 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.004 INFO:tasks.workunit.client.0.smithi090.stdout: "buckets": [ 2021-05-12T12:11:57.005 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.005 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -1, 2021-05-12T12:11:57.005 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default", 2021-05-12T12:11:57.005 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:57.005 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:11:57.006 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.006 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:57.006 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:57.006 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:57.006 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.007 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:11:57.007 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.007 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:57.007 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.008 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.008 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.008 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.008 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -2, 2021-05-12T12:11:57.008 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "default~hdd", 2021-05-12T12:11:57.009 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 11, 2021-05-12T12:11:57.009 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "root", 2021-05-12T12:11:57.009 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.009 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:57.009 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:57.010 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:57.010 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.010 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:11:57.010 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.010 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:57.011 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.011 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.011 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.011 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.011 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -3, 2021-05-12T12:11:57.012 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090", 2021-05-12T12:11:57.012 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:57.012 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:11:57.012 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.012 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.013 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:57.014 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.014 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.014 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:57.014 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.015 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:11:57.015 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.015 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.015 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:57.015 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.016 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:11:57.016 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.016 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.016 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.016 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.017 INFO:tasks.workunit.client.0.smithi090.stdout: "id": -4, 2021-05-12T12:11:57.017 INFO:tasks.workunit.client.0.smithi090.stdout: "name": "smithi090~hdd", 2021-05-12T12:11:57.017 INFO:tasks.workunit.client.0.smithi090.stdout: "type_id": 1, 2021-05-12T12:11:57.017 INFO:tasks.workunit.client.0.smithi090.stdout: "type_name": "host", 2021-05-12T12:11:57.017 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 19206, 2021-05-12T12:11:57.018 INFO:tasks.workunit.client.0.smithi090.stdout: "alg": "straw2", 2021-05-12T12:11:57.018 INFO:tasks.workunit.client.0.smithi090.stdout: "hash": "rjenkins1", 2021-05-12T12:11:57.018 INFO:tasks.workunit.client.0.smithi090.stdout: "items": [ 2021-05-12T12:11:57.018 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.018 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 0, 2021-05-12T12:11:57.019 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.019 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 0 2021-05-12T12:11:57.019 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.019 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.019 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 1, 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 1 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: "id": 2, 2021-05-12T12:11:57.020 INFO:tasks.workunit.client.0.smithi090.stdout: "weight": 6402, 2021-05-12T12:11:57.021 INFO:tasks.workunit.client.0.smithi090.stdout: "pos": 2 2021-05-12T12:11:57.021 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.021 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.021 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.022 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.022 INFO:tasks.workunit.client.0.smithi090.stdout: "rules": [ 2021-05-12T12:11:57.022 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.022 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 0, 2021-05-12T12:11:57.022 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "replicated_rule", 2021-05-12T12:11:57.023 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 0, 2021-05-12T12:11:57.023 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 1, 2021-05-12T12:11:57.023 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 1, 2021-05-12T12:11:57.023 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 10, 2021-05-12T12:11:57.024 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:11:57.024 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.024 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:11:57.024 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:11:57.024 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:11:57.025 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.025 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.025 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_firstn", 2021-05-12T12:11:57.025 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:11:57.025 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:11:57.026 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.026 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.026 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:11:57.026 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.026 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.027 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.027 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.027 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_id": 1, 2021-05-12T12:11:57.027 INFO:tasks.workunit.client.0.smithi090.stdout: "rule_name": "test", 2021-05-12T12:11:57.027 INFO:tasks.workunit.client.0.smithi090.stdout: "ruleset": 1, 2021-05-12T12:11:57.028 INFO:tasks.workunit.client.0.smithi090.stdout: "type": 3, 2021-05-12T12:11:57.028 INFO:tasks.workunit.client.0.smithi090.stdout: "min_size": 3, 2021-05-12T12:11:57.028 INFO:tasks.workunit.client.0.smithi090.stdout: "max_size": 3, 2021-05-12T12:11:57.028 INFO:tasks.workunit.client.0.smithi090.stdout: "steps": [ 2021-05-12T12:11:57.028 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.029 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "set_chooseleaf_tries", 2021-05-12T12:11:57.029 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 5 2021-05-12T12:11:57.029 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.029 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.029 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "set_choose_tries", 2021-05-12T12:11:57.030 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 100 2021-05-12T12:11:57.030 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.030 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.030 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "take", 2021-05-12T12:11:57.030 INFO:tasks.workunit.client.0.smithi090.stdout: "item": -1, 2021-05-12T12:11:57.031 INFO:tasks.workunit.client.0.smithi090.stdout: "item_name": "default" 2021-05-12T12:11:57.031 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.031 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.031 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "choose_indep", 2021-05-12T12:11:57.032 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 0, 2021-05-12T12:11:57.032 INFO:tasks.workunit.client.0.smithi090.stdout: "type": "osd" 2021-05-12T12:11:57.032 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.032 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.032 INFO:tasks.workunit.client.0.smithi090.stdout: "op": "emit" 2021-05-12T12:11:57.033 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.033 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:11:57.033 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.033 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.033 INFO:tasks.workunit.client.0.smithi090.stdout: "tunables": { 2021-05-12T12:11:57.034 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_tries": 0, 2021-05-12T12:11:57.034 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_local_fallback_tries": 0, 2021-05-12T12:11:57.034 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_total_tries": 50, 2021-05-12T12:11:57.034 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_descend_once": 1, 2021-05-12T12:11:57.034 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_vary_r": 1, 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "chooseleaf_stable": 1, 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "straw_calc_version": 1, 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "allowed_bucket_algs": 54, 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "profile": "jewel", 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "optimal_tunables": 1, 2021-05-12T12:11:57.035 INFO:tasks.workunit.client.0.smithi090.stdout: "legacy_tunables": 0, 2021-05-12T12:11:57.036 INFO:tasks.workunit.client.0.smithi090.stdout: "minimum_required_version": "jewel", 2021-05-12T12:11:57.036 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables": 1, 2021-05-12T12:11:57.036 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables2": 1, 2021-05-12T12:11:57.036 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v2_rules": 1, 2021-05-12T12:11:57.036 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables3": 1, 2021-05-12T12:11:57.037 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v3_rules": 0, 2021-05-12T12:11:57.037 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v4_buckets": 1, 2021-05-12T12:11:57.037 INFO:tasks.workunit.client.0.smithi090.stdout: "require_feature_tunables5": 1, 2021-05-12T12:11:57.037 INFO:tasks.workunit.client.0.smithi090.stdout: "has_v5_rules": 0 2021-05-12T12:11:57.037 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.038 INFO:tasks.workunit.client.0.smithi090.stdout: "choose_args": {} 2021-05-12T12:11:57.038 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.038 INFO:tasks.workunit.client.0.smithi090.stdout: "fsmap": { 2021-05-12T12:11:57.038 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 1, 2021-05-12T12:11:57.038 INFO:tasks.workunit.client.0.smithi090.stdout: "default_fscid": -1, 2021-05-12T12:11:57.039 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": { 2021-05-12T12:11:57.039 INFO:tasks.workunit.client.0.smithi090.stdout: "compat": {}, 2021-05-12T12:11:57.039 INFO:tasks.workunit.client.0.smithi090.stdout: "ro_compat": {}, 2021-05-12T12:11:57.039 INFO:tasks.workunit.client.0.smithi090.stdout: "incompat": { 2021-05-12T12:11:57.039 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_1": "base v0.20", 2021-05-12T12:11:57.040 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_2": "client writeable ranges", 2021-05-12T12:11:57.040 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T12:11:57.040 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_4": "dir inode in separate object", 2021-05-12T12:11:57.040 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T12:11:57.040 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T12:11:57.041 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_8": "no anchor table", 2021-05-12T12:11:57.041 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_9": "file layout v2", 2021-05-12T12:11:57.041 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_10": "snaprealm v2" 2021-05-12T12:11:57.041 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.042 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.042 INFO:tasks.workunit.client.0.smithi090.stdout: "feature_flags": { 2021-05-12T12:11:57.042 INFO:tasks.workunit.client.0.smithi090.stdout: "enable_multiple": true, 2021-05-12T12:11:57.042 INFO:tasks.workunit.client.0.smithi090.stdout: "ever_enabled_multiple": true 2021-05-12T12:11:57.042 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.043 INFO:tasks.workunit.client.0.smithi090.stdout: "standbys": [], 2021-05-12T12:11:57.043 INFO:tasks.workunit.client.0.smithi090.stdout: "filesystems": [] 2021-05-12T12:11:57.043 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.043 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_first_committed": 1, 2021-05-12T12:11:57.043 INFO:tasks.workunit.client.0.smithi090.stdout: "mdsmap_last_committed": 1, 2021-05-12T12:11:57.044 INFO:tasks.workunit.client.0.smithi090.stdout: "auth": { 2021-05-12T12:11:57.044 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:11:57.044 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 5, 2021-05-12T12:11:57.044 INFO:tasks.workunit.client.0.smithi090.stdout: "num_secrets": 3 2021-05-12T12:11:57.044 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.045 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:11:57.045 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_active": 1, 2021-05-12T12:11:57.045 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_unknown": 0, 2021-05-12T12:11:57.045 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osd": 3, 2021-05-12T12:11:57.045 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_sum": { 2021-05-12T12:11:57.046 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:11:57.046 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48430, 2021-05-12T12:11:57.046 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 101, 2021-05-12T12:11:57.046 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 1, 2021-05-12T12:11:57.047 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 303, 2021-05-12T12:11:57.047 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:57.047 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:11:57.047 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:57.047 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:57.048 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:57.048 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:57.048 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:11:57.048 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:11:57.048 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:11:57.049 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:11:57.049 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:11:57.049 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:57.049 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:57.049 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:57.050 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:57.050 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:57.050 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:57.050 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:11:57.050 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:57.051 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:57.051 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:11:57.051 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:11:57.051 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:11:57.052 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:11:57.052 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:11:57.052 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:57.052 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:57.052 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:57.053 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:57.053 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:57.053 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:57.053 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:57.053 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:57.054 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:57.054 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:11:57.054 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:11:57.054 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.055 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:11:57.055 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:11:57.055 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:11:57.055 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:57.055 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 0, 2021-05-12T12:11:57.056 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 0, 2021-05-12T12:11:57.056 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:57.056 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:57.056 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:57.056 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:11:57.057 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:11:57.057 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.057 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 18, 2021-05-12T12:11:57.057 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:57.057 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 3, 2021-05-12T12:11:57.058 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 3, 2021-05-12T12:11:57.058 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 0 2021-05-12T12:11:57.058 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.058 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum": { 2021-05-12T12:11:57.058 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:11:57.059 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:11:57.059 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:11:57.059 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:11:57.059 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:57.059 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:57.060 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:11:57.060 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 19336, 2021-05-12T12:11:57.060 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2280, 2021-05-12T12:11:57.060 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 4, 2021-05-12T12:11:57.060 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 17019, 2021-05-12T12:11:57.061 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314553464, 2021-05-12T12:11:57.061 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:11:57.061 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:11:57.061 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322102747136, 2021-05-12T12:11:57.061 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:57.062 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2334720, 2021-05-12T12:11:57.062 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1600163, 2021-05-12T12:11:57.062 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:57.062 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:57.062 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:57.063 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 4866, 2021-05-12T12:11:57.063 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 17427710 2021-05-12T12:11:57.063 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.063 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:11:57.063 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:57.064 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:57.064 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:57.064 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:11:57.064 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:11:57.064 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:11:57.065 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.065 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:11:57.065 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:57.065 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:57.065 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:57.066 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:11:57.066 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.066 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:11:57.066 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:11:57.066 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.067 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_sum_by_class": { 2021-05-12T12:11:57.067 INFO:tasks.workunit.client.0.smithi090.stdout: "hdd": { 2021-05-12T12:11:57.067 INFO:tasks.workunit.client.0.smithi090.stdout: "up_from": 0, 2021-05-12T12:11:57.067 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 0, 2021-05-12T12:11:57.067 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pgs": 3, 2021-05-12T12:11:57.068 INFO:tasks.workunit.client.0.smithi090.stdout: "num_osds": 3, 2021-05-12T12:11:57.068 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_osds": 3, 2021-05-12T12:11:57.068 INFO:tasks.workunit.client.0.smithi090.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T12:11:57.068 INFO:tasks.workunit.client.0.smithi090.stdout: "kb": 314572800, 2021-05-12T12:11:57.068 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used": 19336, 2021-05-12T12:11:57.069 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_data": 2280, 2021-05-12T12:11:57.069 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_omap": 4, 2021-05-12T12:11:57.069 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_used_meta": 17019, 2021-05-12T12:11:57.069 INFO:tasks.workunit.client.0.smithi090.stdout: "kb_avail": 314553464, 2021-05-12T12:11:57.069 INFO:tasks.workunit.client.0.smithi090.stdout: "statfs": { 2021-05-12T12:11:57.070 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 322122547200, 2021-05-12T12:11:57.070 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 322102747136, 2021-05-12T12:11:57.070 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:57.070 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 2334720, 2021-05-12T12:11:57.070 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1600163, 2021-05-12T12:11:57.071 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:57.071 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:57.071 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:57.071 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 4866, 2021-05-12T12:11:57.071 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 17427710 2021-05-12T12:11:57.072 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.072 INFO:tasks.workunit.client.0.smithi090.stdout: "hb_peers": [], 2021-05-12T12:11:57.072 INFO:tasks.workunit.client.0.smithi090.stdout: "snap_trim_queue_len": 0, 2021-05-12T12:11:57.072 INFO:tasks.workunit.client.0.smithi090.stdout: "num_snap_trimming": 0, 2021-05-12T12:11:57.072 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shards_repaired": 0, 2021-05-12T12:11:57.073 INFO:tasks.workunit.client.0.smithi090.stdout: "op_queue_age_hist": { 2021-05-12T12:11:57.073 INFO:tasks.workunit.client.0.smithi090.stdout: "histogram": [], 2021-05-12T12:11:57.073 INFO:tasks.workunit.client.0.smithi090.stdout: "upper_bound": 1 2021-05-12T12:11:57.073 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.073 INFO:tasks.workunit.client.0.smithi090.stdout: "perf_stat": { 2021-05-12T12:11:57.074 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ms": 0, 2021-05-12T12:11:57.074 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ms": 0, 2021-05-12T12:11:57.074 INFO:tasks.workunit.client.0.smithi090.stdout: "commit_latency_ns": 0, 2021-05-12T12:11:57.074 INFO:tasks.workunit.client.0.smithi090.stdout: "apply_latency_ns": 0 2021-05-12T12:11:57.074 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.075 INFO:tasks.workunit.client.0.smithi090.stdout: "alerts": [], 2021-05-12T12:11:57.075 INFO:tasks.workunit.client.0.smithi090.stdout: "network_ping_times": [] 2021-05-12T12:11:57.075 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.075 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.075 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_stats": [ 2021-05-12T12:11:57.076 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.076 INFO:tasks.workunit.client.0.smithi090.stdout: "poolid": 1, 2021-05-12T12:11:57.076 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg": 1, 2021-05-12T12:11:57.076 INFO:tasks.workunit.client.0.smithi090.stdout: "stat_sum": { 2021-05-12T12:11:57.076 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes": 48430, 2021-05-12T12:11:57.077 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects": 101, 2021-05-12T12:11:57.077 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_clones": 1, 2021-05-12T12:11:57.077 INFO:tasks.workunit.client.0.smithi090.stdout: "num_object_copies": 303, 2021-05-12T12:11:57.077 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T12:11:57.077 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_missing": 0, 2021-05-12T12:11:57.078 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_degraded": 101, 2021-05-12T12:11:57.078 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_misplaced": 0, 2021-05-12T12:11:57.078 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_unfound": 0, 2021-05-12T12:11:57.078 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_dirty": 101, 2021-05-12T12:11:57.078 INFO:tasks.workunit.client.0.smithi090.stdout: "num_whiteouts": 0, 2021-05-12T12:11:57.079 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read": 0, 2021-05-12T12:11:57.079 INFO:tasks.workunit.client.0.smithi090.stdout: "num_read_kb": 0, 2021-05-12T12:11:57.079 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write": 101, 2021-05-12T12:11:57.079 INFO:tasks.workunit.client.0.smithi090.stdout: "num_write_kb": 101, 2021-05-12T12:11:57.079 INFO:tasks.workunit.client.0.smithi090.stdout: "num_scrub_errors": 0, 2021-05-12T12:11:57.080 INFO:tasks.workunit.client.0.smithi090.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T12:11:57.080 INFO:tasks.workunit.client.0.smithi090.stdout: "num_deep_scrub_errors": 0, 2021-05-12T12:11:57.080 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_recovered": 0, 2021-05-12T12:11:57.080 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_recovered": 0, 2021-05-12T12:11:57.080 INFO:tasks.workunit.client.0.smithi090.stdout: "num_keys_recovered": 0, 2021-05-12T12:11:57.081 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_omap": 0, 2021-05-12T12:11:57.081 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T12:11:57.081 INFO:tasks.workunit.client.0.smithi090.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T12:11:57.081 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush": 0, 2021-05-12T12:11:57.081 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_kb": 0, 2021-05-12T12:11:57.082 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict": 0, 2021-05-12T12:11:57.082 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_kb": 0, 2021-05-12T12:11:57.082 INFO:tasks.workunit.client.0.smithi090.stdout: "num_promote": 0, 2021-05-12T12:11:57.082 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_high": 0, 2021-05-12T12:11:57.082 INFO:tasks.workunit.client.0.smithi090.stdout: "num_flush_mode_low": 0, 2021-05-12T12:11:57.083 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_some": 0, 2021-05-12T12:11:57.083 INFO:tasks.workunit.client.0.smithi090.stdout: "num_evict_mode_full": 0, 2021-05-12T12:11:57.083 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_pinned": 0, 2021-05-12T12:11:57.083 INFO:tasks.workunit.client.0.smithi090.stdout: "num_legacy_snapsets": 0, 2021-05-12T12:11:57.083 INFO:tasks.workunit.client.0.smithi090.stdout: "num_large_omap_objects": 0, 2021-05-12T12:11:57.084 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_manifest": 0, 2021-05-12T12:11:57.084 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_bytes": 0, 2021-05-12T12:11:57.084 INFO:tasks.workunit.client.0.smithi090.stdout: "num_omap_keys": 0, 2021-05-12T12:11:57.084 INFO:tasks.workunit.client.0.smithi090.stdout: "num_objects_repaired": 0 2021-05-12T12:11:57.084 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.085 INFO:tasks.workunit.client.0.smithi090.stdout: "store_stats": { 2021-05-12T12:11:57.085 INFO:tasks.workunit.client.0.smithi090.stdout: "total": 0, 2021-05-12T12:11:57.085 INFO:tasks.workunit.client.0.smithi090.stdout: "available": 0, 2021-05-12T12:11:57.085 INFO:tasks.workunit.client.0.smithi090.stdout: "internally_reserved": 0, 2021-05-12T12:11:57.085 INFO:tasks.workunit.client.0.smithi090.stdout: "allocated": 1241088, 2021-05-12T12:11:57.086 INFO:tasks.workunit.client.0.smithi090.stdout: "data_stored": 1241088, 2021-05-12T12:11:57.086 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed": 0, 2021-05-12T12:11:57.086 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_allocated": 0, 2021-05-12T12:11:57.086 INFO:tasks.workunit.client.0.smithi090.stdout: "data_compressed_original": 0, 2021-05-12T12:11:57.086 INFO:tasks.workunit.client.0.smithi090.stdout: "omap_allocated": 0, 2021-05-12T12:11:57.087 INFO:tasks.workunit.client.0.smithi090.stdout: "internal_metadata": 0 2021-05-12T12:11:57.087 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.087 INFO:tasks.workunit.client.0.smithi090.stdout: "log_size": 18, 2021-05-12T12:11:57.087 INFO:tasks.workunit.client.0.smithi090.stdout: "ondisk_log_size": 18, 2021-05-12T12:11:57.087 INFO:tasks.workunit.client.0.smithi090.stdout: "up": 3, 2021-05-12T12:11:57.088 INFO:tasks.workunit.client.0.smithi090.stdout: "acting": 3, 2021-05-12T12:11:57.088 INFO:tasks.workunit.client.0.smithi090.stdout: "num_store_stats": 3 2021-05-12T12:11:57.088 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.088 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.088 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_stats": [ 2021-05-12T12:11:57.089 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.089 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:57.089 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 197568495626 2021-05-12T12:11:57.089 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.090 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.090 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:57.090 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 38654705687 2021-05-12T12:11:57.090 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.090 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.091 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:57.091 INFO:tasks.workunit.client.0.smithi090.stdout: "seq": 214748364807 2021-05-12T12:11:57.091 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.091 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.091 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_state": [ 2021-05-12T12:11:57.092 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.092 INFO:tasks.workunit.client.0.smithi090.stdout: "state": "active+undersized+degraded", 2021-05-12T12:11:57.092 INFO:tasks.workunit.client.0.smithi090.stdout: "num": 1 2021-05-12T12:11:57.092 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.092 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.093 INFO:tasks.workunit.client.0.smithi090.stdout: "num_pg_by_osd": [ 2021-05-12T12:11:57.093 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.093 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2, 2021-05-12T12:11:57.093 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:57.093 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:57.094 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:57.094 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.094 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.094 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 2147483647, 2021-05-12T12:11:57.095 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:57.095 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:57.095 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:57.095 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.095 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.096 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 1, 2021-05-12T12:11:57.096 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 0, 2021-05-12T12:11:57.096 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 0, 2021-05-12T12:11:57.096 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:57.096 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.097 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.097 INFO:tasks.workunit.client.0.smithi090.stdout: "osd": 0, 2021-05-12T12:11:57.097 INFO:tasks.workunit.client.0.smithi090.stdout: "num_primary_pg": 1, 2021-05-12T12:11:57.097 INFO:tasks.workunit.client.0.smithi090.stdout: "num_acting_pg": 1, 2021-05-12T12:11:57.098 INFO:tasks.workunit.client.0.smithi090.stdout: "num_up_not_acting_pg": 0 2021-05-12T12:11:57.098 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.098 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.098 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": [ 2021-05-12T12:11:57.098 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:11:57.099 INFO:tasks.workunit.client.0.smithi090.stdout: "pool": 1, 2021-05-12T12:11:57.099 INFO:tasks.workunit.client.0.smithi090.stdout: "purged_snaps": {} 2021-05-12T12:11:57.099 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.099 INFO:tasks.workunit.client.0.smithi090.stdout: ], 2021-05-12T12:11:57.100 INFO:tasks.workunit.client.0.smithi090.stdout: "servicemap": { 2021-05-12T12:11:57.100 INFO:tasks.workunit.client.0.smithi090.stdout: "epoch": 5, 2021-05-12T12:11:57.100 INFO:tasks.workunit.client.0.smithi090.stdout: "modified": "2021-05-12T12:10:25.537855+0000", 2021-05-12T12:11:57.100 INFO:tasks.workunit.client.0.smithi090.stdout: "services": {} 2021-05-12T12:11:57.101 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:11:57.101 INFO:tasks.workunit.client.0.smithi090.stdout: "paxos": { 2021-05-12T12:11:57.101 INFO:tasks.workunit.client.0.smithi090.stdout: "first_committed": 1, 2021-05-12T12:11:57.101 INFO:tasks.workunit.client.0.smithi090.stdout: "last_committed": 531, 2021-05-12T12:11:57.101 INFO:tasks.workunit.client.0.smithi090.stdout: "last_pn": 0, 2021-05-12T12:11:57.102 INFO:tasks.workunit.client.0.smithi090.stdout: "accepted_pn": 0 2021-05-12T12:11:57.102 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:11:57.102 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:11:57.103 INFO:tasks.workunit.client.0.smithi090.stderr:report 860950609 2021-05-12T12:11:57.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T12:11:57.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2021-05-12T12:11:57.104 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery 2021-05-12T12:11:57.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:57.190 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:57.191 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:23.126154+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:11:57.191 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:11:57.191 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:57.192 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:11:57.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:57.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T12:11:57.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:11:57.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:11:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:11:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:11:57.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.0.asok 2021-05-12T12:11:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:11:57.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T12:11:57.297 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:11:57.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T12:11:57.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2021-05-12T12:11:57.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T12:11:57.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T12:11:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:11:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:11:57.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.2.asok 2021-05-12T12:11:57.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:11:57.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T12:11:57.401 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:11:57.409 INFO:tasks.workunit.client.0.smithi090.stdout:revive divergent 1 2021-05-12T12:11:57.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2021-05-12T12:11:57.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:11:57.751 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:11:57.752 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:23.126154+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:11:57.752 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:11:57.752 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:11:57.753 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:11:57.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2021-05-12T12:11:58.416 INFO:tasks.workunit.client.0.smithi090.stderr:noup is set 2021-05-12T12:11:58.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T12:11:58.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:11:58.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:11:58.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:11:58.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:11:58.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:11:58.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:11:58.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:11:58.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:11:58.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:11:58.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:11:58.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:11:58.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:11:58.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:11:58.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:11:58.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:11:58.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:11:58.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:11:58.432 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:11:58.432 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:11:58.433 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:11:58.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:58.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:11:58.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:11:58.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:11:58.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:11:58.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:11:58.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:11:58.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:11:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:11:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:11:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:11:58.436 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:11:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:11:58.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:11:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:11:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:11:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:11:58.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:11:58.455 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:58.452+0000 7f5d951370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:58.457 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:58.453+0000 7f5d951370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:58.458 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:58.455+0000 7f5d951370c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:11:58.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T12:11:58.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2021-05-12T12:11:58.974 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:11:58.971+0000 7f5d951370c0 -1 Falling back to public interface 2021-05-12T12:12:01.199 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:01.196+0000 7f5d951370c0 -1 osd.1 29 log_to_monitors {default=true} 2021-05-12T12:12:03.796 INFO:tasks.workunit.client.0.smithi090.stdout:delay recovery divergent 2021-05-12T12:12:03.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2021-05-12T12:12:03.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:04.148 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:04.148 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:23.126154+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:04.148 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:04.149 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:04.149 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:04.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2021-05-12T12:12:04.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:12:04.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:12:04.160 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:12:04.161 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:04.161 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:12:04.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29659/ceph-osd.1.asok 2021-05-12T12:12:04.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T12:12:04.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29659/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T12:12:04.255 INFO:tasks.workunit.client.0.smithi090.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T12:12:04.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2021-05-12T12:12:04.917 INFO:tasks.workunit.client.0.smithi090.stderr:noup is unset 2021-05-12T12:12:04.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2021-05-12T12:12:04.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:04.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:12:04.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:04.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:04.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:04.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:04.929 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:04.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:04.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:05.374 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [36,43) [v2:127.0.0.1:6802/79511,v1:127.0.0.1:6803/79511] [v2:127.0.0.1:6804/79511,v1:127.0.0.1:6805/79511] exists,up 01e065f1-a569-44af-90fd-e9233051a6cf 2021-05-12T12:12:05.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:05.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:05.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:05.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2021-05-12T12:12:05.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:05.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:12:05.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:05.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:05.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:05.377 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:05.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:05.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:05.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:05.753 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6818/82329,v1:127.0.0.1:6819/82329] [v2:127.0.0.1:6820/82329,v1:127.0.0.1:6821/82329] exists,up 219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:12:05.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:05.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:05.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:05.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2021-05-12T12:12:05.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:05.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:12:05.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:05.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:05.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:05.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:05.757 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:05.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:05.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:12:06.114 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [38,43) [v2:127.0.0.1:6810/80194,v1:127.0.0.1:6811/80194] [v2:127.0.0.1:6812/80194,v1:127.0.0.1:6813/80194] exists,up 79c730d8-a2c9-4bb0-a0cd-79bdc776a37a 2021-05-12T12:12:06.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:06.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:06.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:06.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:06.473 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:06.474 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:23.126154+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:06.474 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:06.474 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:06.475 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:06.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2021-05-12T12:12:06.485 INFO:tasks.workunit.client.0.smithi090.stdout:wait for peering 2021-05-12T12:12:06.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:06.827 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:06.827 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T12:11:23.126154+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:06.827 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:06.828 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:06.828 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:06.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2021-05-12T12:12:19.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2021-05-12T12:12:19.227 INFO:tasks.workunit.client.0.smithi090.stdout:killing divergent 1 2021-05-12T12:12:19.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:19.580 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:19.580 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T12:12:05.508936+0000 43'102 57:128 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:19.581 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:19.581 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:19.582 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:19.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T12:12:19.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:19.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:19.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:19.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:19.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:19.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:19.697 INFO:tasks.workunit.client.0.smithi090.stdout:reviving divergent 1 2021-05-12T12:12:19.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2021-05-12T12:12:19.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:20.039 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:20.039 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T12:12:05.508936+0000 43'102 57:128 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:20.039 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:20.040 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:20.040 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:20.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T12:12:20.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T12:12:20.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:12:20.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:12:20.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:12:20.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T12:12:20.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T12:12:20.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:20.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:20.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:20.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T12:12:20.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T12:12:20.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:20.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:12:20.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T12:12:20.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:12:20.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:20.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:20.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:20.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:20.057 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:12:20.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:12:20.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' 2021-05-12T12:12:20.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:20.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T12:12:20.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T12:12:20.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:20.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:20.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:20.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:12:20.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:12:20.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T12:12:20.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:12:20.061 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:12:20.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8160268e-296f-4dfd-917e-d0d3cee77a07 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29659/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:20.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T12:12:20.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:12:20.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:12:20.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:12:20.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:12:20.081 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:20.078+0000 7f2d51f470c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:20.082 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:20.079+0000 7f2d51f470c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:20.084 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:20.081+0000 7f2d51f470c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:20.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:12:20.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:20.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:12:20.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:20.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:20.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:20.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:20.421 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:20.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:20.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:20.598 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:20.596+0000 7f2d51f470c0 -1 Falling back to public interface 2021-05-12T12:12:20.784 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6818/82329,v1:127.0.0.1:6819/82329] [v2:127.0.0.1:6820/82329,v1:127.0.0.1:6821/82329] exists,up 219c9c7a-4f27-4908-a6a5-bc27283f6f3a 2021-05-12T12:12:20.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:20.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:20.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:20.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2021-05-12T12:12:23.106 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:23.104+0000 7f2d51f470c0 -1 osd.1 57 log_to_monitors {default=true} 2021-05-12T12:12:40.786 INFO:tasks.workunit.client.0.smithi090.stdout:allowing recovery 2021-05-12T12:12:40.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2021-05-12T12:12:40.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:41.139 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:41.139 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T12:12:31.266203+0000 57'103 62:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:41.139 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:41.139 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:41.140 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:41.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:41.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T12:12:41.274 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:41.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:41.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T12:12:41.406 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:41.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T12:12:41.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T12:12:41.539 INFO:tasks.workunit.client.0.smithi090.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T12:12:41.548 INFO:tasks.workunit.client.0.smithi090.stdout:reading divergent objects 2021-05-12T12:12:41.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2021-05-12T12:12:41.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T12:12:41.887 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:12:41.887 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T12:12:31.266203+0000 57'103 62:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T12:09:27.931477+0000 0'0 2021-05-12T12:09:27.931477+0000 0 2021-05-12T12:12:41.888 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:12:41.888 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:12:41.889 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:12:41.899 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2021-05-12T12:12:41.900 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2021-05-12T12:12:41.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:41.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T12:12:41.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:41.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T12:12:41.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:41.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T12:12:42.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T12:12:42.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T12:12:42.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2021-05-12T12:12:42.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T12:12:42.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:12:42.051 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T12:10:03.241743+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.051 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T12:10:03.241743+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.051 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:42.052 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2021-05-12T12:12:42.052 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2021-05-12T12:12:42.052 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2021-05-12T12:12:42.052 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T12:10:10.497099+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.053 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T12:10:10.497099+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.053 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:42.053 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2021-05-12T12:12:42.053 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2021-05-12T12:12:42.054 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:12:42.054 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2021-05-12T12:12:42.054 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T12:10:10.497099+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.054 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2021-05-12T12:12:42.055 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4554.0:1 2021-05-12T12:10:09.284532+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T12:10:10.292708+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.055 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4554.0:1 2021-05-12T12:10:09.284532+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.055 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T12:10:10.292708+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.056 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T12:12:42.056 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2021-05-12T12:12:42.056 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T12:12:42.056 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T12:12:42.057 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2021-05-12T12:12:42.057 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T12:10:10.292708+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T12:12:42.057 INFO:tasks.workunit.client.0.smithi090.stdout:td/divergent-priors/osd.1.log:2021-05-12T12:12:05.372+0000 7f5d7102c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T12:12:42.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T12:12:42.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T12:12:42.059 INFO:tasks.workunit.client.0.smithi090.stdout:success 2021-05-12T12:12:42.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2021-05-12T12:12:42.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2021-05-12T12:12:42.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T12:12:42.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T12:12:42.540 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test' does not exist 2021-05-12T12:12:42.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2021-05-12T12:12:42.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:42.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:42.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:42.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:42.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:42.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:42.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T12:12:42.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:12:42.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:12:42.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:12:42.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:42.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:42.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:42.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:42.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:42.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:42.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:42.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:42.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:42.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:42.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:42.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:42.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:42.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:42.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:42.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:12:42.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:12:42.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:42.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:42.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:12:42.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:12:42.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:12:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/divergent-priors 0 2021-05-12T12:12:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T12:12:42.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:12:42.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T12:12:42.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:42.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:42.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:42.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:42.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:42.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:42.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:42.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:42.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:42.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:42.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:42.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:42.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:42.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:42.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:42.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:12:42.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T12:12:42.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:42.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:42.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29659 2021-05-12T12:12:42.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29659 2021-05-12T12:12:42.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:42.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:42.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:12:42.696 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:12:42.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:12:42.730 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2021-05-12T12:12:42.730 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2021-05-12T12:12:42.790 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:12:42.803 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:12:42.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:12:42.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:12:42.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:12:42.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:12:42.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:12:42.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:12:42.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:12:42.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:12:42.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/ec-error-rollforward 2021-05-12T12:12:42.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2021-05-12T12:12:42.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2021-05-12T12:12:42.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2021-05-12T12:12:42.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2021-05-12T12:12:42.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2021-05-12T12:12:42.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2021-05-12T12:12:42.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none ' 2021-05-12T12:12:42.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:42.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2021-05-12T12:12:42.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2021-05-12T12:12:42.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2021-05-12T12:12:42.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2021-05-12T12:12:42.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2021-05-12T12:12:42.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2021-05-12T12:12:42.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2021-05-12T12:12:42.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:12:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2021-05-12T12:12:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2021-05-12T12:12:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2021-05-12T12:12:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2021-05-12T12:12:42.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2021-05-12T12:12:42.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:12:42.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:12:42.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:12:42.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:12:42.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:12:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:12:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:12:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:12:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:12:42.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:12:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:12:42.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:12:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:12:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:12:42.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:12:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:12:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:12:42.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:12:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:12:42.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:12:42.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:12:42.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:12:42.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:42.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:42.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83890 2021-05-12T12:12:42.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:12:42.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:12:42.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2021-05-12T12:12:42.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:12:42.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:42.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:42.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83890 2021-05-12T12:12:42.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:12:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:12:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:12:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:12:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2021-05-12T12:12:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2021-05-12T12:12:42.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2021-05-12T12:12:42.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2021-05-12T12:12:42.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2021-05-12T12:12:42.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:12:42.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:12:42.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:12:42.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2021-05-12T12:12:42.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2021-05-12T12:12:43.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:12:43.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:43.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:43.196 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:43.197 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:43.197 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:43.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:43.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:12:43.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:12:43.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:12:43.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:12:43.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:12:43.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:12:43.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:12:43.441 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:12:43.442 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:12:43.442 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:12:43.442 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:12:43.442 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:43.443 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:43.443 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83890/ceph-mon.a.asok 2021-05-12T12:12:43.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:12:43.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.83890/ceph-mon.a.asok config get fsid 2021-05-12T12:12:43.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:12:43.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:12:43.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:12:43.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:12:43.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:12:43.535 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:43.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83890/ceph-mon.a.asok 2021-05-12T12:12:43.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:12:43.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.83890/ceph-mon.a.asok config get mon_host 2021-05-12T12:12:43.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2021-05-12T12:12:43.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2021-05-12T12:12:43.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:12:43.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:12:43.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:12:43.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2021-05-12T12:12:43.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:12:44.100 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:12:44.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:44.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:44.101 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:44.102 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:44.102 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:44.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:44.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:12:44.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:12:44.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2021-05-12T12:12:44.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:44.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:44.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:12:44.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:44.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T12:12:44.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:44.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:44.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T12:12:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T12:12:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:44.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:44.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:44.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:44.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:44.129 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:44.130 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:44.130 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:44.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:44.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:44.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:44.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:44.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:44.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:44.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:44.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:44.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:44.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:44.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:44.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:44.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T12:12:44.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:44.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=028c7861-2c33-424a-8577-38b044945a8f 2021-05-12T12:12:44.133 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 028c7861-2c33-424a-8577-38b044945a8f 2021-05-12T12:12:44.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 028c7861-2c33-424a-8577-38b044945a8f' 2021-05-12T12:12:44.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:44.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA8xptg4lwVCBAAMMf740yiar/RzCHoOM69pw== 2021-05-12T12:12:44.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA8xptg4lwVCBAAMMf740yiar/RzCHoOM69pw=="}' 2021-05-12T12:12:44.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 028c7861-2c33-424a-8577-38b044945a8f -i td/ec-error-rollforward/0/new.json 2021-05-12T12:12:44.422 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/0/new.json 2021-05-12T12:12:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8xptg4lwVCBAAMMf740yiar/RzCHoOM69pw== --osd-uuid 028c7861-2c33-424a-8577-38b044945a8f 2021-05-12T12:12:44.454 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.451+0000 7f2d270e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:44.456 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.453+0000 7f2d270e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:44.457 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.454+0000 7f2d270e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:44.458 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.455+0000 7f2d270e60c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:44.459 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.455+0000 7f2d270e60c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:44.459 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.455+0000 7f2d270e60c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T12:12:44.498 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:44.494+0000 7f2d270e60c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2021-05-12T12:12:49.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2021-05-12T12:12:49.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:12:49.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:12:49.069 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:12:49.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:12:49.542 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:12:49.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:12:49.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:49.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:12:49.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:12:49.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:12:49.561 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:49.557+0000 7f35550610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:49.562 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:49.559+0000 7f35550610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:49.564 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:49.561+0000 7f35550610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:49.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:12:49.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:49.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:12:49.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:49.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:49.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:49.903 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:49.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:49.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:49.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:50.079 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:50.075+0000 7f35550610c0 -1 Falling back to public interface 2021-05-12T12:12:50.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:12:51.284 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:12:51.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:51.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:51.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:12:52.254 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:52.250+0000 7f35550610c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:12:52.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:12:52.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:52.647 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:12:52.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:12:52.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:52.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:12:53.036 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85010,v1:127.0.0.1:6803/85010] [v2:127.0.0.1:6804/85010,v1:127.0.0.1:6805/85010] exists,up 028c7861-2c33-424a-8577-38b044945a8f 2021-05-12T12:12:53.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:12:53.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:12:53.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:12:53.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2021-05-12T12:12:53.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:12:53.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:12:53.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:12:53.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:12:53.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T12:12:53.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:12:53.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:12:53.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:12:53.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:12:53.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T12:12:53.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T12:12:53.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:12:53.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:12:53.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:12:53.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:12:53.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:12:53.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:12:53.045 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:12:53.046 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:12:53.046 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:12:53.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:53.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:12:53.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:12:53.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:12:53.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:12:53.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:12:53.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:12:53.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:12:53.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:12:53.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:12:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:12:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T12:12:53.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:12:53.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0add37de-7947-4a0e-9d03-fbda9d79eeb0 2021-05-12T12:12:53.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 0add37de-7947-4a0e-9d03-fbda9d79eeb0' 2021-05-12T12:12:53.050 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 0add37de-7947-4a0e-9d03-fbda9d79eeb0 2021-05-12T12:12:53.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:12:53.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBFxptgBd/tAhAAtfiFwewXBJqEd1qR0KdFXA== 2021-05-12T12:12:53.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBFxptgBd/tAhAAtfiFwewXBJqEd1qR0KdFXA=="}' 2021-05-12T12:12:53.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0add37de-7947-4a0e-9d03-fbda9d79eeb0 -i td/ec-error-rollforward/1/new.json 2021-05-12T12:12:53.575 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:12:53.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/1/new.json 2021-05-12T12:12:53.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBFxptgBd/tAhAAtfiFwewXBJqEd1qR0KdFXA== --osd-uuid 0add37de-7947-4a0e-9d03-fbda9d79eeb0 2021-05-12T12:12:53.609 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.605+0000 7f44d4f4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:53.611 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.607+0000 7f44d4f4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:53.612 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.609+0000 7f44d4f4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:53.613 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.609+0000 7f44d4f4e0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:53.613 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.609+0000 7f44d4f4e0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:53.613 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.609+0000 7f44d4f4e0c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T12:12:53.703 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:53.700+0000 7f44d4f4e0c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2021-05-12T12:12:58.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2021-05-12T12:12:58.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:12:58.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:12:58.307 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:12:58.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:12:58.789 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:12:58.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:12:58.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:12:58.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:12:58.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:12:58.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:12:58.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:58.804+0000 7f6e30aa00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:58.810 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:58.806+0000 7f6e30aa00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:58.810 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:58.807+0000 7f6e30aa00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:12:59.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:12:59.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:12:59.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:12:59.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:12:59.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:12:59.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:12:59.169 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:12:59.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:12:59.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:12:59.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:12:59.330 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:12:59.327+0000 7f6e30aa00c0 -1 Falling back to public interface 2021-05-12T12:12:59.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:00.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:00.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:00.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:00.524 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:13:00.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:00.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:13:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:01.534 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:01.531+0000 7f6e30aa00c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:13:01.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:01.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:01.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:01.886 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:13:01.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:01.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:13:02.240 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/86403,v1:127.0.0.1:6811/86403] [v2:127.0.0.1:6812/86403,v1:127.0.0.1:6813/86403] exists,up 0add37de-7947-4a0e-9d03-fbda9d79eeb0 2021-05-12T12:13:02.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:02.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:02.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:02.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2021-05-12T12:13:02.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:02.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:13:02.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:13:02.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:13:02.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T12:13:02.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:02.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:02.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:02.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:02.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T12:13:02.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T12:13:02.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:02.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:13:02.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:02.248 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:13:02.248 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:02.248 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:02.248 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:02.249 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:02.249 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:02.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:02.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:02.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:02.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:13:02.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:13:02.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:02.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:02.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:02.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:02.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:02.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:13:02.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:13:02.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T12:13:02.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:13:02.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e4dbdd0c-0870-4a3a-8bb6-3eec0699183f 2021-05-12T12:13:02.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 e4dbdd0c-0870-4a3a-8bb6-3eec0699183f' 2021-05-12T12:13:02.253 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 e4dbdd0c-0870-4a3a-8bb6-3eec0699183f 2021-05-12T12:13:02.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:13:02.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBOxptgRdIuDxAAchYfEjJhUHop8S15FnFo7A== 2021-05-12T12:13:02.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBOxptgRdIuDxAAchYfEjJhUHop8S15FnFo7A=="}' 2021-05-12T12:13:02.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e4dbdd0c-0870-4a3a-8bb6-3eec0699183f -i td/ec-error-rollforward/2/new.json 2021-05-12T12:13:02.736 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:13:02.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/2/new.json 2021-05-12T12:13:02.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOxptgRdIuDxAAchYfEjJhUHop8S15FnFo7A== --osd-uuid e4dbdd0c-0870-4a3a-8bb6-3eec0699183f 2021-05-12T12:13:02.769 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.766+0000 7fe142b3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:02.770 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.768+0000 7fe142b3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:02.772 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.769+0000 7fe142b3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:02.773 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.770+0000 7fe142b3c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:13:02.773 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.770+0000 7fe142b3c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:13:02.773 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.770+0000 7fe142b3c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T12:13:02.826 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:02.823+0000 7fe142b3c0c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2021-05-12T12:13:07.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2021-05-12T12:13:07.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:13:07.463 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:13:07.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:13:07.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:13:07.943 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:13:07.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:13:07.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:07.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:13:07.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:13:07.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:13:07.962 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:07.960+0000 7fc6ff1cf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:07.964 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:07.961+0000 7fc6ff1cf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:07.965 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:07.962+0000 7fc6ff1cf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:08.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:13:08.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:08.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:13:08.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:08.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:08.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:08.299 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:08.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:08.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:08.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:13:08.482 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:08.479+0000 7fc6ff1cf0c0 -1 Falling back to public interface 2021-05-12T12:13:08.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:09.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:09.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:09.657 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:13:09.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:09.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:09.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:13:10.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:10.683 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:10.680+0000 7fc6ff1cf0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:13:11.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:11.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:11.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:11.014 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:13:11.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:11.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:13:11.407 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/87788,v1:127.0.0.1:6819/87788] [v2:127.0.0.1:6820/87788,v1:127.0.0.1:6821/87788] exists,up e4dbdd0c-0870-4a3a-8bb6-3eec0699183f 2021-05-12T12:13:11.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:11.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2021-05-12T12:13:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:13:11.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:13:11.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:13:11.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T12:13:11.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:11.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:11.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:11.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:11.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T12:13:11.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T12:13:11.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:11.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:13:11.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:11.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:13:11.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:11.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:11.416 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:11.416 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:11.416 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:11.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:11.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:11.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:11.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:13:11.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:13:11.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:11.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:11.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:11.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:11.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:11.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:13:11.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:13:11.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T12:13:11.420 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:13:11.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e22a9b12-2f72-4838-8dac-8a1c0919afe1 2021-05-12T12:13:11.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 e22a9b12-2f72-4838-8dac-8a1c0919afe1' 2021-05-12T12:13:11.420 INFO:tasks.workunit.client.0.smithi090.stdout:add osd3 e22a9b12-2f72-4838-8dac-8a1c0919afe1 2021-05-12T12:13:11.421 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:13:11.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBXxptgnBUiGRAAY9zD+BIBogcNuDMPn/4X9Q== 2021-05-12T12:13:11.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBXxptgnBUiGRAAY9zD+BIBogcNuDMPn/4X9Q=="}' 2021-05-12T12:13:11.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e22a9b12-2f72-4838-8dac-8a1c0919afe1 -i td/ec-error-rollforward/3/new.json 2021-05-12T12:13:11.892 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:13:11.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/3/new.json 2021-05-12T12:13:11.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBXxptgnBUiGRAAY9zD+BIBogcNuDMPn/4X9Q== --osd-uuid e22a9b12-2f72-4838-8dac-8a1c0919afe1 2021-05-12T12:13:11.925 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.922+0000 7f8abf5570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:11.926 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.924+0000 7f8abf5570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:11.927 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.925+0000 7f8abf5570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:11.928 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.925+0000 7f8abf5570c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:13:11.929 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.925+0000 7f8abf5570c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:13:11.929 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.925+0000 7f8abf5570c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T12:13:11.980 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:11.977+0000 7f8abf5570c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2021-05-12T12:13:16.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2021-05-12T12:13:16.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:13:16.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:13:16.717 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd3 key to auth repository 2021-05-12T12:13:16.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:13:17.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:13:17.198 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:13:17.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:17.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:13:17.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:13:17.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:13:17.217 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:17.214+0000 7ff547e830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.218 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:17.216+0000 7ff547e830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.219 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:17.217+0000 7ff547e830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:17.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:13:17.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:17.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:13:17.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:17.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:17.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:17.556 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:17.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:17.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:17.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:17.733+0000 7ff547e830c0 -1 Falling back to public interface 2021-05-12T12:13:17.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:18.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:18.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:18.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:18.911 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:13:18.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:18.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:19.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:19.909 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:19.906+0000 7ff547e830c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:13:20.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:20.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:20.275 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:13:20.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:20.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:20.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:20.640 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/89173,v1:127.0.0.1:6827/89173] [v2:127.0.0.1:6828/89173,v1:127.0.0.1:6829/89173] exists,up e22a9b12-2f72-4838-8dac-8a1c0919afe1 2021-05-12T12:13:20.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:20.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:20.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:20.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2021-05-12T12:13:21.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2021-05-12T12:13:22.092 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'ec' already exists 2021-05-12T12:13:22.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2021-05-12T12:13:23.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2021-05-12T12:13:23.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 87788 2021-05-12T12:13:23.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=89786 2021-05-12T12:13:23.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2021-05-12T12:13:23.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2021-05-12T12:13:24.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 89807' 2021-05-12T12:13:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2021-05-12T12:13:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 89808' 2021-05-12T12:13:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2021-05-12T12:13:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 89809' 2021-05-12T12:13:24.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2021-05-12T12:13:24.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2021-05-12T12:13:25.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2021-05-12T12:13:25.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:25.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:25.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:25.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:25.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:25.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:25.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 89786 89807 89808 89809 2021-05-12T12:13:25.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2021-05-12T12:13:25.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2021-05-12T12:13:25.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:25.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:25.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:13:25.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:25.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T12:13:25.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:25.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:25.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:25.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:25.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T12:13:25.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T12:13:25.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:25.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:25.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:25.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:25.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:25.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:25.931 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:25.932 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:25.932 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:25.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:25.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:25.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:25.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:25.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:25.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:25.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:25.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:25.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:25.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:25.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T12:13:25.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:13:25.935 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:13:25.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:25.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/0/whoami 2021-05-12T12:13:25.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:13:25.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:25.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:25.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:25.949 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:25.946+0000 7ff093a3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:25.951 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:25.948+0000 7ff093a3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:25.953 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:25.950+0000 7ff093a3d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:26.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:13:26.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:26.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:13:26.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:26.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:26.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:26.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:26.296 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:26.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:26.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:26.473 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:26.470+0000 7ff093a3d0c0 -1 Falling back to public interface 2021-05-12T12:13:26.664 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85010,v1:127.0.0.1:6803/85010] [v2:127.0.0.1:6804/85010,v1:127.0.0.1:6805/85010] exists,up 028c7861-2c33-424a-8577-38b044945a8f 2021-05-12T12:13:26.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:26.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:26.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:26.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2021-05-12T12:13:26.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:26.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:26.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:13:26.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:26.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T12:13:26.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:26.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:26.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:26.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:26.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T12:13:26.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T12:13:26.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:26.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:26.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:26.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:26.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:26.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:26.672 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:26.673 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:26.673 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:26.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:26.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:26.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:26.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:26.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:26.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:26.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:26.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:26.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:26.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:26.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T12:13:26.676 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:13:26.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:13:26.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:26.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/1/whoami 2021-05-12T12:13:26.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:13:26.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:26.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:26.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:26.685 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:26.682+0000 7fcf5eca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:26.686 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:26.684+0000 7fcf5eca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:26.688 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:26.686+0000 7fcf5eca50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:27.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:13:27.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:27.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:13:27.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:27.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:27.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:27.030 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:27.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:27.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:27.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:13:27.208 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:27.206+0000 7fcf5eca50c0 -1 Falling back to public interface 2021-05-12T12:13:27.405 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/86403,v1:127.0.0.1:6811/86403] [v2:127.0.0.1:6812/86403,v1:127.0.0.1:6813/86403] exists,up 0add37de-7947-4a0e-9d03-fbda9d79eeb0 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:27.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:27.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:13:27.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:27.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T12:13:27.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:27.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:27.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:27.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:27.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T12:13:27.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T12:13:27.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:27.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:27.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:27.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:27.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:27.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:27.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:27.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:27.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:27.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:27.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:27.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:27.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:27.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:27.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:27.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:27.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:27.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:27.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:13:27.417 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:13:27.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/2/whoami 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:27.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:27.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:27.426 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:27.423+0000 7f1aae0730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:27.427 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:27.425+0000 7f1aae0730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:27.429 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:27.426+0000 7f1aae0730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:27.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:13:27.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:27.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:13:27.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:27.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:27.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:27.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:27.768 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:27.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:27.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:13:27.944 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:27.941+0000 7f1aae0730c0 -1 Falling back to public interface 2021-05-12T12:13:28.121 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/87788,v1:127.0.0.1:6819/87788] [v2:127.0.0.1:6820/87788,v1:127.0.0.1:6821/87788] exists,up e4dbdd0c-0870-4a3a-8bb6-3eec0699183f 2021-05-12T12:13:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2021-05-12T12:13:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T12:13:28.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:13:28.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:13:28.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:13:28.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T12:13:28.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T12:13:28.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:28.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:28.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:28.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T12:13:28.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T12:13:28.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:28.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:13:28.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T12:13:28.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:13:28.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:28.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:28.129 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:28.130 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:28.130 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:28.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:28.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' 2021-05-12T12:13:28.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:28.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T12:13:28.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T12:13:28.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:28.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:28.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:28.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:13:28.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:13:28.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T12:13:28.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:13:28.133 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:13:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=9b9ee1b0-fc72-4a8c-9ddc-993d0850ba08 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83890/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:28.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/3/whoami 2021-05-12T12:13:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:13:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:13:28.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:13:28.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:13:28.142 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:28.139+0000 7f248fd2a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:28.143 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:28.141+0000 7f248fd2a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:28.145 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:28.142+0000 7f248fd2a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:28.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:13:28.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:28.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:13:28.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:28.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:28.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:28.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:28.487 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:28.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:28.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:13:28.661 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:28.659+0000 7f248fd2a0c0 -1 Falling back to public interface 2021-05-12T12:13:28.850 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/89173,v1:127.0.0.1:6827/89173] [v2:127.0.0.1:6828/89173,v1:127.0.0.1:6829/89173] exists,up e22a9b12-2f72-4838-8dac-8a1c0919afe1 2021-05-12T12:13:28.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:28.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:28.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:28.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2021-05-12T12:13:28.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:13:28.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:13:28.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:13:28.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:13:28.854 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:13:28.855 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:13:28.855 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:13:28.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:13:28.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:13:28.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:13:28.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:13:28.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:13:28.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:13:28.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:13:28.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:13:29.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:13:29.293 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:13:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:13:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:3' 2021-05-12T12:13:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:13:29.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:29.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:13:29.429 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:29.426+0000 7ff093a3d0c0 -1 osd.0 24 log_to_monitors {default=true} 2021-05-12T12:13:30.588 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:30.585+0000 7fcf5eca50c0 -1 osd.1 24 log_to_monitors {default=true} 2021-05-12T12:13:31.486 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:31.483+0000 7f1aae0730c0 -1 osd.2 23 log_to_monitors {default=true} 2021-05-12T12:13:32.172 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:32.169+0000 7f248fd2a0c0 -1 osd.3 24 log_to_monitors {default=true} 2021-05-12T12:13:34.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=115964116995 2021-05-12T12:13:34.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 115964116995 2021-05-12T12:13:34.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116995' 2021-05-12T12:13:34.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:34.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:13:34.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=128849018883 2021-05-12T12:13:34.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 128849018883 2021-05-12T12:13:34.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116995 1-128849018883' 2021-05-12T12:13:34.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:34.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:13:34.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=146028888067 2021-05-12T12:13:34.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 146028888067 2021-05-12T12:13:34.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-146028888067' 2021-05-12T12:13:34.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:13:34.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:13:34.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=154618822659 2021-05-12T12:13:34.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 154618822659 2021-05-12T12:13:34.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-146028888067 3-154618822659' 2021-05-12T12:13:34.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:34.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-115964116995 2021-05-12T12:13:34.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:34.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:13:34.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-115964116995 2021-05-12T12:13:34.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:34.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=115964116995 2021-05-12T12:13:34.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 115964116995' 2021-05-12T12:13:34.805 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 115964116995 2021-05-12T12:13:34.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:13:35.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 115964116995 -lt 115964116995 2021-05-12T12:13:35.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:35.163 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-128849018883 2021-05-12T12:13:35.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:35.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:13:35.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-128849018883 2021-05-12T12:13:35.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:35.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=128849018883 2021-05-12T12:13:35.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 128849018883' 2021-05-12T12:13:35.167 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 128849018883 2021-05-12T12:13:35.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:13:35.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 128849018883 -lt 128849018883 2021-05-12T12:13:35.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:35.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-146028888067 2021-05-12T12:13:35.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:35.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:13:35.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-146028888067 2021-05-12T12:13:35.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:35.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=146028888067 2021-05-12T12:13:35.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2021-05-12T12:13:35.538 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 146028888067 2021-05-12T12:13:35.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:13:35.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 146028888067 -lt 146028888067 2021-05-12T12:13:35.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:13:35.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-154618822659 2021-05-12T12:13:35.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:13:35.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:13:35.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-154618822659 2021-05-12T12:13:35.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:13:35.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=154618822659 2021-05-12T12:13:35.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2021-05-12T12:13:35.894 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 154618822659 2021-05-12T12:13:35.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:13:36.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 154618822659 2021-05-12T12:13:36.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:13:37.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:13:37.254 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:13:37.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 154618822659 -lt 154618822659 2021-05-12T12:13:37.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:13:37.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:13:37.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:13:38.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 1 == 0 2021-05-12T12:13:38.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:13:38.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:13:38.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:13:38.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:13:38.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:13:38.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:13:38.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:13:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:13:38.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:13:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:13:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:13:38.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 1 2021-05-12T12:13:38.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:13:38.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:13:38.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2021-05-12T12:13:38.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:13:38.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:13:38.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:13:38.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:38.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:38.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:38.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:38.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:39.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:39.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:39.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:39.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:39.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:39.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:39.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:39.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:39.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:39.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:39.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:13:39.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:13:39.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:39.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:39.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83890 2021-05-12T12:13:39.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:39.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:39.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:13:39.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/ec-error-rollforward 0 2021-05-12T12:13:39.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T12:13:39.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:13:39.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T12:13:39.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:39.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:39.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:39.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:39.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:39.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:39.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:39.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:39.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:39.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:39.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:39.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:39.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:39.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:39.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:13:39.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T12:13:39.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:39.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83890 2021-05-12T12:13:39.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83890 2021-05-12T12:13:39.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:39.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:39.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:13:39.059 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:13:39.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:13:39.121 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2021-05-12T12:13:39.122 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2021-05-12T12:13:39.181 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:13:39.189 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:13:39.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:13:39.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:13:39.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:13:39.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:13:39.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:13:39.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:13:39.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:13:39.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:13:39.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bench 2021-05-12T12:13:39.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2021-05-12T12:13:39.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2021-05-12T12:13:39.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2021-05-12T12:13:39.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2021-05-12T12:13:39.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2021-05-12T12:13:39.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2021-05-12T12:13:39.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=0ade6592-96bb-4142-88c0-ce23ecaf2ff6 --auth-supported=none ' 2021-05-12T12:13:39.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2021-05-12T12:13:39.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2021-05-12T12:13:39.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2021-05-12T12:13:39.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:13:39.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2021-05-12T12:13:39.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2021-05-12T12:13:39.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2021-05-12T12:13:39.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2021-05-12T12:13:39.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2021-05-12T12:13:39.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:13:39.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:13:39.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:13:39.198 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:13:39.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:13:39.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:13:39.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:13:39.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:13:39.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:13:39.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:13:39.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:13:39.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:13:39.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:13:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:13:39.201 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:13:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:13:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:13:39.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:13:39.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:13:39.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:13:39.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:13:39.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:13:39.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:39.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92449 2021-05-12T12:13:39.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:13:39.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:13:39.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2021-05-12T12:13:39.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:13:39.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92449 2021-05-12T12:13:39.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:13:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:13:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:13:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:13:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2021-05-12T12:13:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2021-05-12T12:13:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2021-05-12T12:13:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2021-05-12T12:13:39.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2021-05-12T12:13:39.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:13:39.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:13:39.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:13:39.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2021-05-12T12:13:39.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2021-05-12T12:13:39.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:13:39.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:39.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:39.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92449/$cluster-$name.asok' 2021-05-12T12:13:39.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:13:39.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:13:39.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:13:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:13:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:13:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:13:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:13:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:13:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:13:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:13:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:13:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.789 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:39.789 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92449/ceph-mon.a.asok 2021-05-12T12:13:39.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:13:39.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.92449/ceph-mon.a.asok config get fsid 2021-05-12T12:13:39.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:13:39.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:13:39.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:13:39.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:13:39.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:13:39.881 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:13:39.881 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:13:39.882 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:13:39.882 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:13:39.882 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:39.882 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:39.882 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92449/ceph-mon.a.asok 2021-05-12T12:13:39.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:13:39.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.92449/ceph-mon.a.asok config get mon_host 2021-05-12T12:13:39.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2021-05-12T12:13:39.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2021-05-12T12:13:39.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:13:39.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:13:39.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:13:39.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2021-05-12T12:13:39.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:13:40.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:13:40.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:40.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:40.854 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:40.855 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:40.858 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:40.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92449/$cluster-$name.asok' 2021-05-12T12:13:40.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:13:40.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92449/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:13:40.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2021-05-12T12:13:40.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2021-05-12T12:13:40.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:13:40.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:13:40.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:13:40.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2021-05-12T12:13:40.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ade6592-96bb-4142-88c0-ce23ecaf2ff6 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2021-05-12T12:13:40.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:13:40.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:13:40.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:13:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2021-05-12T12:13:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2021-05-12T12:13:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:13:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:13:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2021-05-12T12:13:40.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:13:40.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:13:40.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:13:40.880 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:13:40.880 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:13:40.881 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:13:40.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92449/$cluster-$name.asok' 2021-05-12T12:13:40.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92449/$cluster-$name.asok' 2021-05-12T12:13:40.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:13:40.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:13:40.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:13:40.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2021-05-12T12:13:40.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2021-05-12T12:13:40.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:13:40.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:13:40.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:40.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:13:40.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:13:40.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bench/0 2021-05-12T12:13:40.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:13:40.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=200d7a1b-cb75-4820-a4cf-d7d407ce33db 2021-05-12T12:13:40.885 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 200d7a1b-cb75-4820-a4cf-d7d407ce33db 2021-05-12T12:13:40.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 200d7a1b-cb75-4820-a4cf-d7d407ce33db' 2021-05-12T12:13:40.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:13:40.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB0xptg5oSJNBAAdtHD2vQeDl8edoEBhZ8Lyg== 2021-05-12T12:13:40.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB0xptg5oSJNBAAdtHD2vQeDl8edoEBhZ8Lyg=="}' 2021-05-12T12:13:40.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 200d7a1b-cb75-4820-a4cf-d7d407ce33db -i td/osd-bench/0/new.json 2021-05-12T12:13:41.192 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:41.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bench/0/new.json 2021-05-12T12:13:41.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=0ade6592-96bb-4142-88c0-ce23ecaf2ff6 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB0xptg5oSJNBAAdtHD2vQeDl8edoEBhZ8Lyg== --osd-uuid 200d7a1b-cb75-4820-a4cf-d7d407ce33db 2021-05-12T12:13:41.224 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.220+0000 7f4700cc50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:41.226 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.222+0000 7f4700cc50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:41.228 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.224+0000 7f4700cc50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:41.229 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.225+0000 7f4700cc50c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:41.229 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.225+0000 7f4700cc50c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:41.230 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.225+0000 7f4700cc50c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T12:13:41.267 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:41.263+0000 7f4700cc50c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2021-05-12T12:13:45.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bench/0/keyring 2021-05-12T12:13:45.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:13:45.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:13:45.838 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:13:45.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:13:46.306 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:13:46.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:13:46.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:13:46.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=0ade6592-96bb-4142-88c0-ce23ecaf2ff6 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:13:46.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:13:46.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:13:46.325 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:46.322+0000 7f8d1017f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:46.327 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:46.323+0000 7f8d1017f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:46.329 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:46.325+0000 7f8d1017f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:13:46.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:13:46.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:13:46.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:13:46.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:13:46.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:13:46.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:46.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:13:46.666 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:13:46.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:46.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:46.845 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:46.842+0000 7f8d1017f0c0 -1 Falling back to public interface 2021-05-12T12:13:47.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:48.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:48.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:48.046 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:13:48.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:13:48.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:48.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:48.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:13:49.044 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:13:49.040+0000 7f8d1017f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:13:49.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:13:49.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:13:49.409 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:13:49.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:13:49.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:13:49.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:13:49.768 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/93569,v1:127.0.0.1:6803/93569] [v2:127.0.0.1:6804/93569,v1:127.0.0.1:6805/93569] exists,up 200d7a1b-cb75-4820-a4cf-d7d407ce33db 2021-05-12T12:13:49.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:13:49.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:13:49.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:13:49.770 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:49.770 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2021-05-12T12:13:49.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2021-05-12T12:13:49.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:49.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2021-05-12T12:13:49.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2021-05-12T12:13:49.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:49.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2021-05-12T12:13:49.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2021-05-12T12:13:49.799 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2021-05-12T12:13:49.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2021-05-12T12:13:49.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2021-05-12T12:13:49.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2021-05-12T12:13:49.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:49.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:49.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_max_block_size 2021-05-12T12:13:49.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:49.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:49.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 1024 67108865 2021-05-12T12:13:49.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:49.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:49.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2021-05-12T12:13:49.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:49.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2021-05-12T12:13:49.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2021-05-12T12:13:49.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2021-05-12T12:13:49.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:49.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:49.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=bench_small_size_max_iops 2021-05-12T12:13:49.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:49.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:49.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3072001 1024 2021-05-12T12:13:50.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:50.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:50.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2021-05-12T12:13:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2021-05-12T12:13:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2021-05-12T12:13:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2021-05-12T12:13:50.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T12:13:50.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T12:13:50.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_large_size_max_throughput 2021-05-12T12:13:50.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T12:13:50.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T12:13:50.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2021-05-12T12:13:50.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T12:13:50.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T12:13:50.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2021-05-12T12:13:50.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T12:13:50.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2021-05-12T12:15:02.366 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:15:02.367 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_written": 1073741824, 2021-05-12T12:15:02.367 INFO:tasks.workunit.client.0.smithi090.stdout: "blocksize": 4194304, 2021-05-12T12:15:02.368 INFO:tasks.workunit.client.0.smithi090.stdout: "elapsed_sec": 37.433458275, 2021-05-12T12:15:02.368 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_per_sec": 28684013.539756235, 2021-05-12T12:15:02.368 INFO:tasks.workunit.client.0.smithi090.stdout: "iops": 6.8388017510786616 2021-05-12T12:15:02.368 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:15:02.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2021-05-12T12:15:03.320 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:15:03.320 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_written": 10, 2021-05-12T12:15:03.321 INFO:tasks.workunit.client.0.smithi090.stdout: "blocksize": 4444, 2021-05-12T12:15:03.321 INFO:tasks.workunit.client.0.smithi090.stdout: "elapsed_sec": 0.050423145000000003, 2021-05-12T12:15:03.321 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_per_sec": 198.32162392885252, 2021-05-12T12:15:03.321 INFO:tasks.workunit.client.0.smithi090.stdout: "iops": 0.044626828066798498 2021-05-12T12:15:03.321 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:15:03.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2021-05-12T12:15:03.612 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:15:03.613 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_written": 1, 2021-05-12T12:15:03.613 INFO:tasks.workunit.client.0.smithi090.stdout: "blocksize": 14456, 2021-05-12T12:15:03.613 INFO:tasks.workunit.client.0.smithi090.stdout: "elapsed_sec": 0.051061259999999997, 2021-05-12T12:15:03.614 INFO:tasks.workunit.client.0.smithi090.stdout: "bytes_per_sec": 19.584318914182692, 2021-05-12T12:15:03.614 INFO:tasks.workunit.client.0.smithi090.stdout: "iops": 0.0013547536603612819 2021-05-12T12:15:03.614 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:15:03.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2021-05-12T12:15:03.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:15:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:15:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:15:03.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:15:03.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:15:03.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:15:03.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:15:03.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:15:03.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:15:03.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:15:03.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:15:03.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:15:03.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:15:03.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:15:03.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:15:03.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:03.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:15:03.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:15:03.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:03.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:15:03.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:15:03.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:15:04.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:15:04.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:15:04.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92449 2021-05-12T12:15:04.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:15:04.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:15:04.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:15:04.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bench 0 2021-05-12T12:15:04.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T12:15:04.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:15:04.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T12:15:04.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:15:04.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:15:04.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:15:04.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:15:04.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:15:04.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:15:04.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:15:04.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:15:04.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:15:04.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:15:04.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:15:04.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:15:04.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:04.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:15:04.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:15:04.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:04.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:15:04.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:15:04.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T12:15:04.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:15:04.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92449 2021-05-12T12:15:04.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92449 2021-05-12T12:15:04.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:15:04.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:15:04.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:15:04.150 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:15:04.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:15:04.211 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2021-05-12T12:15:04.211 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2021-05-12T12:15:04.271 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:15:04.281 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:15:04.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:15:04.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:15:04.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:15:04.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:15:04.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:15:04.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:15:04.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:15:04.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:15:04.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bluefs-volume-ops 2021-05-12T12:15:04.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2021-05-12T12:15:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T12:15:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T12:15:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2021-05-12T12:15:04.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2021-05-12T12:15:04.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none ' 2021-05-12T12:15:04.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T12:15:04.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2021-05-12T12:15:04.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2021-05-12T12:15:04.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2021-05-12T12:15:04.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2021-05-12T12:15:04.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2021-05-12T12:15:04.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2021-05-12T12:15:04.286 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2021-05-12T12:15:04.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:15:04.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2021-05-12T12:15:04.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2021-05-12T12:15:04.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2021-05-12T12:15:04.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2021-05-12T12:15:04.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:15:04.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:15:04.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:15:04.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:15:04.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:15:04.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:15:04.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:15:04.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:15:04.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:15:04.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:15:04.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:15:04.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:15:04.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:15:04.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:15:04.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:04.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:15:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:15:04.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:15:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:15:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:15:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:15:04.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:15:04.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:04.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94350 2021-05-12T12:15:04.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:15:04.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:15:04.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2021-05-12T12:15:04.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:15:04.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:04.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94350 2021-05-12T12:15:04.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:15:04.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:15:04.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:15:04.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:15:04.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2021-05-12T12:15:04.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2021-05-12T12:15:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2021-05-12T12:15:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2021-05-12T12:15:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T12:15:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2021-05-12T12:15:04.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:15:04.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:15:04.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:15:04.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2021-05-12T12:15:04.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:04.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:15:04.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:04.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:04.657 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:04.658 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.658 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:04.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:04.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:15:04.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:15:04.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:15:04.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:15:04.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:15:04.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:15:04.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:15:04.911 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:15:04.911 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:15:04.911 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:15:04.912 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:15:04.912 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:04.912 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:04.912 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94350/ceph-mon.a.asok 2021-05-12T12:15:04.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:15:04.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.94350/ceph-mon.a.asok config get fsid 2021-05-12T12:15:05.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:15:05.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:15:05.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:15:05.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:15:05.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:15:05.006 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:15:05.006 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:15:05.006 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:15:05.007 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:15:05.007 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:05.007 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:05.007 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94350/ceph-mon.a.asok 2021-05-12T12:15:05.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:15:05.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.94350/ceph-mon.a.asok config get mon_host 2021-05-12T12:15:05.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2021-05-12T12:15:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:15:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:15:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:15:05.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2021-05-12T12:15:05.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:15:05.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:15:05.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:05.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:05.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:05.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:05.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:05.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:05.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:15:05.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:15:06.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:15:06.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:06.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:06.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:15:06.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:06.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:15:06.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:06.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:06.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:06.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:06.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:15:06.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:15:06.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:06.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:06.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:06.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:06.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:06.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:06.018 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:06.018 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:06.018 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:06.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:06.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:06.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:06.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:06.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:06.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:06.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:06.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:06.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:06.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:06.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:06.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:06.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:15:06.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:06.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:15:06.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 64254159-a6cd-4e09-9277-b2d729d7b59e' 2021-05-12T12:15:06.022 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:15:06.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:06.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDKxptgRBNfARAABfAcJVJWaPoSa+25lVSQfw== 2021-05-12T12:15:06.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDKxptgRBNfARAABfAcJVJWaPoSa+25lVSQfw=="}' 2021-05-12T12:15:06.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 64254159-a6cd-4e09-9277-b2d729d7b59e -i td/osd-bluefs-volume-ops/0/new.json 2021-05-12T12:15:06.332 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:15:06.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2021-05-12T12:15:06.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKxptgRBNfARAABfAcJVJWaPoSa+25lVSQfw== --osd-uuid 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:15:06.365 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.362+0000 7f022a7460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:06.367 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.364+0000 7f022a7460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:06.368 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.365+0000 7f022a7460c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:06.369 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.366+0000 7f022a7460c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:15:06.369 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.366+0000 7f022a7460c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:15:06.369 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.366+0000 7f022a7460c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T12:15:06.407 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:06.404+0000 7f022a7460c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2021-05-12T12:15:15.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2021-05-12T12:15:15.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:15.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:15:15.133 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:15:15.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:15.623 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:15:15.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:15:15.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:15.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:15.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:15.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:15.643 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:15.640+0000 7fccea0e70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:15.645 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:15.642+0000 7fccea0e70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:15.646 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:15.643+0000 7fccea0e70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:15.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:15:15.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:15.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:15:15.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:15.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:15.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:15.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:15.985 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:15:15.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:15.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:16.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:16.661 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:16.659+0000 7fccea0e70c0 -1 Falling back to public interface 2021-05-12T12:15:17.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:17.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:17.337 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:15:17.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:17.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:17.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:17.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:18.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:18.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:18.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:18.697 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:15:18.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:18.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:19.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:20.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:20.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:20.068 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:15:20.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:20.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:20.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:20.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:21.106 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:21.104+0000 7fccea0e70c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:15:21.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:21.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:21.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:21.436 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:15:21.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:21.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:15:21.803 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95867,v1:127.0.0.1:6803/95867] [v2:127.0.0.1:6804/95867,v1:127.0.0.1:6805/95867] exists,up 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:15:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:21.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:15:21.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=95867 2021-05-12T12:15:21.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:15:21.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:21.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:15:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:15:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:21.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:21.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:21.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:15:21.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:15:21.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:21.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:21.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:21.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:21.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:21.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:21.813 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:21.813 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:21.813 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:21.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:21.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:21.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:21.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:21.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:21.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:21.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:21.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:21.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:21.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:21.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:21.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:21.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:15:21.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:21.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:15:21.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 5e29472e-2a43-441d-ac2f-cced9184cbe1' 2021-05-12T12:15:21.817 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:15:21.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:21.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDZxptgGcS+MBAARSTweT9hIRb+SJ5Z2SpajA== 2021-05-12T12:15:21.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDZxptgGcS+MBAARSTweT9hIRb+SJ5Z2SpajA=="}' 2021-05-12T12:15:21.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5e29472e-2a43-441d-ac2f-cced9184cbe1 -i td/osd-bluefs-volume-ops/1/new.json 2021-05-12T12:15:22.315 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:15:22.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2021-05-12T12:15:22.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZxptgGcS+MBAARSTweT9hIRb+SJ5Z2SpajA== --osd-uuid 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:15:22.349 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.346+0000 7f4f5bfbd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:22.350 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.348+0000 7f4f5bfbd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:22.351 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.349+0000 7f4f5bfbd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:22.352 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.350+0000 7f4f5bfbd0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:22.353 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.350+0000 7f4f5bfbd0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:22.353 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.350+0000 7f4f5bfbd0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T12:15:22.394 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:22.391+0000 7f4f5bfbd0c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2021-05-12T12:15:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2021-05-12T12:15:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:15:30.729 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:15:30.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:31.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:15:31.209 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:15:31.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:31.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:31.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:31.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:31.228 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:31.226+0000 7fce79b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.241 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:31.238+0000 7fce79b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.243 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:31.240+0000 7fce79b970c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:31.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:15:31.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:31.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:15:31.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:31.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:31.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:31.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:31.568 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:15:31.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:31.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:31.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:32.259 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:32.256+0000 7fce79b970c0 -1 Falling back to public interface 2021-05-12T12:15:32.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:32.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:32.942 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:15:32.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:32.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:32.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:33.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:34.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:34.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:34.305 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:15:34.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:34.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:34.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:34.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:35.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:35.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:35.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:35.682 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:15:35.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:35.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:36.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:36.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:36.805+0000 7fce79b970c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:15:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:37.046 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:15:37.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:37.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:37.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:15:37.477 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/98087,v1:127.0.0.1:6811/98087] [v2:127.0.0.1:6812/98087,v1:127.0.0.1:6813/98087] exists,up 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:15:37.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:37.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:37.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:37.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:15:37.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=98087 2021-05-12T12:15:37.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:15:37.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:37.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:37.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:15:37.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:37.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:15:37.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:37.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:37.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:37.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:37.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:15:37.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:15:37.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:37.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:37.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:37.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:37.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:37.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:37.487 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:37.487 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:37.487 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:37.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:37.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:37.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:37.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:37.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:37.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:37.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:37.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:37.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:37.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:37.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:37.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:37.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:15:37.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:37.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:15:37.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 d54e3313-a346-4e73-be73-a214d6572386' 2021-05-12T12:15:37.492 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:15:37.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:37.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDpxptgyBB1HRAA0hl+KLLpmRYjt7uXe0T7qA== 2021-05-12T12:15:37.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDpxptgyBB1HRAA0hl+KLLpmRYjt7uXe0T7qA=="}' 2021-05-12T12:15:37.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d54e3313-a346-4e73-be73-a214d6572386 -i td/osd-bluefs-volume-ops/2/new.json 2021-05-12T12:15:38.094 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:15:38.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2021-05-12T12:15:38.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpxptgyBB1HRAA0hl+KLLpmRYjt7uXe0T7qA== --osd-uuid d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:15:38.128 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.125+0000 7faea0ed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:38.144 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.141+0000 7faea0ed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:38.145 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.142+0000 7faea0ed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:38.146 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.144+0000 7faea0ed30c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:38.147 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.144+0000 7faea0ed30c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:38.147 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.144+0000 7faea0ed30c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T12:15:38.202 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:38.200+0000 7faea0ed30c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2021-05-12T12:15:46.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2021-05-12T12:15:46.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:15:46.788 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:15:46.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:15:46.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:15:47.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:15:47.273 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:15:47.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:15:47.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:47.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:15:47.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:15:47.292 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:47.288+0000 7f17efb300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:47.294 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:47.290+0000 7f17efb300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:47.295 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:47.291+0000 7f17efb300c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:47.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:15:47.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:15:47.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:15:47.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:15:47.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:15:47.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:47.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:15:47.630 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:15:47.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:47.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:48.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:48.311 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:48.308+0000 7f17efb300c0 -1 Falling back to public interface 2021-05-12T12:15:49.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:49.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:49.006 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:15:49.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:15:49.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:49.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:49.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:50.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:50.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:50.371 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:15:50.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:15:50.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:50.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:50.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:51.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:51.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:51.739 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:15:51.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:15:51.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:51.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:52.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:52.822 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:52.818+0000 7f17efb300c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:15:53.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:53.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:53.100 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:15:53.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:15:53.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:53.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:53.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:15:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:15:54.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:15:54.492 INFO:tasks.workunit.client.0.smithi090.stdout:5 2021-05-12T12:15:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:15:54.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:15:54.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:15:54.879 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/100307,v1:127.0.0.1:6819/100307] [v2:127.0.0.1:6820/100307,v1:127.0.0.1:6821/100307] exists,up d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:15:54.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:15:54.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:15:54.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:15:54.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:15:54.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=100307 2021-05-12T12:15:54.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:15:54.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:15:54.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:15:54.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:15:54.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:15:54.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:15:54.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:15:54.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:15:54.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:15:54.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:15:54.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:15:54.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:15:54.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:15:54.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:15:54.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:15:54.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:15:54.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:15:54.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:15:54.890 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:15:54.890 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:15:54.890 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:15:54.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:15:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:15:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:15:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:15:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:15:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:15:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:15:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:15:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:15:54.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:15:54.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:15:54.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:15:54.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:15:54.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:15:54.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 d71baae8-2593-4a1c-95b4-3821273338a2' 2021-05-12T12:15:54.894 INFO:tasks.workunit.client.0.smithi090.stdout:add osd3 d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:15:54.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:15:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD6xptgkg1qNRAAEd471CY5SxTQWs4x0mo89g== 2021-05-12T12:15:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD6xptgkg1qNRAAEd471CY5SxTQWs4x0mo89g=="}' 2021-05-12T12:15:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d71baae8-2593-4a1c-95b4-3821273338a2 -i td/osd-bluefs-volume-ops/3/new.json 2021-05-12T12:15:55.355 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:15:55.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2021-05-12T12:15:55.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD6xptgkg1qNRAAEd471CY5SxTQWs4x0mo89g== --osd-uuid d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:15:55.388 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.385+0000 7f8b8e0ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:55.390 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.387+0000 7f8b8e0ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:55.391 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.388+0000 7f8b8e0ac0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:15:55.392 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.388+0000 7f8b8e0ac0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:55.392 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.388+0000 7f8b8e0ac0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:55.392 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.388+0000 7f8b8e0ac0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T12:15:55.434 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:15:55.431+0000 7f8b8e0ac0c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2021-05-12T12:16:03.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2021-05-12T12:16:03.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:16:03.864 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd3 key to auth repository 2021-05-12T12:16:03.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:16:03.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:16:04.346 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:16:04.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:16:04.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:16:04.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:16:04.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:16:04.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:16:04.366 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:16:04.363+0000 7f49cb4270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:16:04.368 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:16:04.365+0000 7f49cb4270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:16:04.368 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:16:04.366+0000 7f49cb4270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:16:04.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:16:04.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:16:04.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:16:04.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:16:04.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:16:04.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:04.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:16:04.727 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:16:04.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:04.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:05.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:05.383 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:16:05.381+0000 7f49cb4270c0 -1 Falling back to public interface 2021-05-12T12:16:06.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:06.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:06.109 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:16:06.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:16:06.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:06.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:06.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:07.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:07.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:07.465 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:16:07.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:16:07.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:07.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:07.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:08.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:08.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:08.839 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:16:08.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:16:08.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:08.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:09.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:16:09.837 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:16:09.834+0000 7f49cb4270c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:16:10.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:16:10.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:16:10.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:16:10.207 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:16:10.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:16:10.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:16:10.616 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/102561,v1:127.0.0.1:6827/102561] [v2:127.0.0.1:6828/102561,v1:127.0.0.1:6829/102561] exists,up d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:16:10.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:16:10.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:16:10.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:16:10.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:16:10.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=102561 2021-05-12T12:16:10.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2021-05-12T12:16:15.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2021-05-12T12:16:15.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T12:16:16.189 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'foo' already exists 2021-05-12T12:16:16.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:16:17.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:16:17.228 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:16:17.229 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:16:17.229 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_103571 2021-05-12T12:16:17.229 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:16:17.229 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:16:18.229 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 38 22 0.0859311 0.0859375 0.326094 0.54524 2021-05-12T12:16:19.229 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 87 71 0.138658 0.191406 0.343723 0.422425 2021-05-12T12:16:20.229 INFO:tasks.workunit.client.0.smithi090.stdout: 3 16 137 121 0.157534 0.195312 0.241434 0.370892 2021-05-12T12:16:21.230 INFO:tasks.workunit.client.0.smithi090.stdout: 4 16 193 177 0.172832 0.21875 0.260255 0.348781 2021-05-12T12:16:22.229 INFO:tasks.workunit.client.0.smithi090.stdout: 5 16 247 231 0.180448 0.210938 0.301371 0.336357 2021-05-12T12:16:23.229 INFO:tasks.workunit.client.0.smithi090.stdout: 6 16 298 282 0.183572 0.199219 0.308164 0.329852 2021-05-12T12:16:24.229 INFO:tasks.workunit.client.0.smithi090.stdout: 7 16 346 330 0.184129 0.1875 0.316431 0.330817 2021-05-12T12:16:25.229 INFO:tasks.workunit.client.0.smithi090.stdout: 8 16 401 385 0.187965 0.214844 0.289699 0.32624 2021-05-12T12:16:26.230 INFO:tasks.workunit.client.0.smithi090.stdout: 9 16 455 439 0.190515 0.210938 0.248733 0.32168 2021-05-12T12:16:27.230 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16 506 490 0.191383 0.199219 0.301989 0.319988 2021-05-12T12:16:28.230 INFO:tasks.workunit.client.0.smithi090.stdout: 11 16 561 545 0.193513 0.214844 0.284235 0.318934 2021-05-12T12:16:29.230 INFO:tasks.workunit.client.0.smithi090.stdout: 12 16 609 593 0.19301 0.1875 0.384249 0.320416 2021-05-12T12:16:30.230 INFO:tasks.workunit.client.0.smithi090.stdout: 13 16 663 647 0.194387 0.210938 0.276668 0.31841 2021-05-12T12:16:31.230 INFO:tasks.workunit.client.0.smithi090.stdout: 14 16 711 695 0.193893 0.1875 0.3048 0.317176 2021-05-12T12:16:32.231 INFO:tasks.workunit.client.0.smithi090.stdout: 15 16 769 753 0.196069 0.226562 0.292849 0.315553 2021-05-12T12:16:33.231 INFO:tasks.workunit.client.0.smithi090.stdout: 16 16 823 807 0.196997 0.210938 0.273317 0.31463 2021-05-12T12:16:34.231 INFO:tasks.workunit.client.0.smithi090.stdout: 17 16 881 865 0.198734 0.226562 0.273386 0.312235 2021-05-12T12:16:35.231 INFO:tasks.workunit.client.0.smithi090.stdout: 18 16 929 913 0.198109 0.1875 0.284623 0.311637 2021-05-12T12:16:36.231 INFO:tasks.workunit.client.0.smithi090.stdout: 19 16 986 970 0.199399 0.222656 0.303934 0.310301 2021-05-12T12:16:37.231 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:16:37.229337+0000 min lat: 0.240989 max lat: 0.772656 avg lat: 0.309501 2021-05-12T12:16:37.232 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:16:37.232 INFO:tasks.workunit.client.0.smithi090.stdout: 20 16 1041 1025 0.200171 0.214844 0.320523 0.309501 2021-05-12T12:16:38.231 INFO:tasks.workunit.client.0.smithi090.stdout: 21 16 1095 1079 0.200682 0.210938 0.293845 0.309265 2021-05-12T12:16:39.231 INFO:tasks.workunit.client.0.smithi090.stdout: 22 16 1143 1127 0.200081 0.1875 0.307029 0.308206 2021-05-12T12:16:40.232 INFO:tasks.workunit.client.0.smithi090.stdout: 23 16 1185 1169 0.198514 0.164062 0.307327 0.312488 2021-05-12T12:16:41.232 INFO:tasks.workunit.client.0.smithi090.stdout: 24 16 1239 1223 0.19903 0.210938 0.323394 0.311772 2021-05-12T12:16:42.232 INFO:tasks.workunit.client.0.smithi090.stdout: 25 16 1297 1281 0.200131 0.226562 0.319533 0.310556 2021-05-12T12:16:43.232 INFO:tasks.workunit.client.0.smithi090.stdout: 26 16 1351 1335 0.200545 0.210938 0.302588 0.309456 2021-05-12T12:16:44.232 INFO:tasks.workunit.client.0.smithi090.stdout: 27 16 1409 1393 0.201508 0.226562 0.284477 0.308598 2021-05-12T12:16:45.232 INFO:tasks.workunit.client.0.smithi090.stdout: 28 16 1463 1447 0.201844 0.210938 0.258796 0.307938 2021-05-12T12:16:46.232 INFO:tasks.workunit.client.0.smithi090.stdout: 29 16 1521 1505 0.202695 0.226562 0.273199 0.307237 2021-05-12T12:16:47.232 INFO:tasks.workunit.client.0.smithi090.stdout: 30 16 1575 1559 0.202969 0.210938 0.29215 0.306537 2021-05-12T12:16:48.233 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 30.2282 2021-05-12T12:16:48.233 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 1575 2021-05-12T12:16:48.233 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:16:48.233 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:16:48.234 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.20353 2021-05-12T12:16:48.234 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.0269363 2021-05-12T12:16:48.234 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.226562 2021-05-12T12:16:48.234 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0.0859375 2021-05-12T12:16:48.235 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 52 2021-05-12T12:16:48.235 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 6.89569 2021-05-12T12:16:48.235 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 58 2021-05-12T12:16:48.235 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 22 2021-05-12T12:16:48.236 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.306183 2021-05-12T12:16:48.236 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 0.0612466 2021-05-12T12:16:48.236 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 0.772656 2021-05-12T12:16:48.236 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.240989 2021-05-12T12:16:48.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2021-05-12T12:16:48.386 INFO:tasks.workunit.client.0.smithi090.stdout:after bench 2021-05-12T12:16:48.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 95867 2021-05-12T12:16:48.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2021-05-12T12:16:49.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 95867 2021-05-12T12:16:49.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (95867) - No such process 2021-05-12T12:16:49.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2021-05-12T12:16:49.767 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:16:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98087 2021-05-12T12:16:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2021-05-12T12:16:50.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98087 2021-05-12T12:16:50.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (98087) - No such process 2021-05-12T12:16:50.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2021-05-12T12:16:51.304 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:16:51.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 100307 2021-05-12T12:16:51.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2021-05-12T12:16:52.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 100307 2021-05-12T12:16:52.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (100307) - No such process 2021-05-12T12:16:52.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2021-05-12T12:16:52.704 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:16:52.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 102561 2021-05-12T12:16:52.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2021-05-12T12:16:53.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 102561 2021-05-12T12:16:53.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (102561) - No such process 2021-05-12T12:16:53.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2021-05-12T12:16:54.250 INFO:tasks.workunit.client.0.smithi090.stderr:osd.3 is already down. 2021-05-12T12:16:54.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:16:56.039 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:16:56.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:16:57.824 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:16:57.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:16:59.611 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:16:59.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:01.397 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:01.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2021-05-12T12:17:01.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2021-05-12T12:17:01.418 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:02.217 INFO:tasks.workunit.client.0.smithi090.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:02.217 INFO:tasks.workunit.client.0.smithi090.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:02.217 INFO:tasks.workunit.client.0.smithi090.stdout:2 : device size 0x100000000 : using 0x804cf000(2.0 GiB) 2021-05-12T12:17:02.218 INFO:tasks.workunit.client.0.smithi090.stdout:Expanding DB/WAL... 2021-05-12T12:17:02.218 INFO:tasks.workunit.client.0.smithi090.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:02.242 INFO:tasks.workunit.client.0.smithi090.stdout:0 : size label updated to 536870912 2021-05-12T12:17:02.242 INFO:tasks.workunit.client.0.smithi090.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:02.267 INFO:tasks.workunit.client.0.smithi090.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:02.267 INFO:tasks.workunit.client.0.smithi090.stdout:2 : expanding from 0x80000000 to 0x100000000 2021-05-12T12:17:02.427 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:02.424+0000 7fcac1fd9400 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T12:17:02.427 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:02.424+0000 7fcac1fd9400 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T12:17:07.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2021-05-12T12:17:07.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2021-05-12T12:17:07.194 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:07.993 INFO:tasks.workunit.client.0.smithi090.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:07.994 INFO:tasks.workunit.client.0.smithi090.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:07.994 INFO:tasks.workunit.client.0.smithi090.stdout:2 : device size 0x101000000 : using 0x814df000(2.0 GiB) 2021-05-12T12:17:07.994 INFO:tasks.workunit.client.0.smithi090.stdout:Expanding DB/WAL... 2021-05-12T12:17:07.994 INFO:tasks.workunit.client.0.smithi090.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:08.016 INFO:tasks.workunit.client.0.smithi090.stdout:0 : size label updated to 536870912 2021-05-12T12:17:08.017 INFO:tasks.workunit.client.0.smithi090.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:08.033 INFO:tasks.workunit.client.0.smithi090.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:08.033 INFO:tasks.workunit.client.0.smithi090.stdout:2 : expanding from 0x80000000 to 0x101000000 2021-05-12T12:17:08.185 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:08.182+0000 7f6de7a6a400 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T12:17:08.186 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:08.182+0000 7f6de7a6a400 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T12:17:12.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2021-05-12T12:17:12.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2021-05-12T12:17:12.917 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:13.704 INFO:tasks.workunit.client.0.smithi090.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:13.704 INFO:tasks.workunit.client.0.smithi090.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:13.705 INFO:tasks.workunit.client.0.smithi090.stdout:2 : device size 0x100020000 : using 0x80519000(2.0 GiB) 2021-05-12T12:17:13.705 INFO:tasks.workunit.client.0.smithi090.stdout:Expanding DB/WAL... 2021-05-12T12:17:13.705 INFO:tasks.workunit.client.0.smithi090.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:13.732 INFO:tasks.workunit.client.0.smithi090.stdout:0 : size label updated to 536870912 2021-05-12T12:17:13.732 INFO:tasks.workunit.client.0.smithi090.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:13.748 INFO:tasks.workunit.client.0.smithi090.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:13.749 INFO:tasks.workunit.client.0.smithi090.stdout:2 : expanding from 0x80000000 to 0x100020000 2021-05-12T12:17:13.926 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:13.923+0000 7fa544ecc400 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T12:17:13.926 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:13.923+0000 7fa544ecc400 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T12:17:18.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2021-05-12T12:17:18.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2021-05-12T12:17:18.627 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:19.413 INFO:tasks.workunit.client.0.smithi090.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-05-12T12:17:19.413 INFO:tasks.workunit.client.0.smithi090.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T12:17:19.414 INFO:tasks.workunit.client.0.smithi090.stdout:2 : device size 0xfff00000 : using 0x803d9000(2.0 GiB) 2021-05-12T12:17:19.414 INFO:tasks.workunit.client.0.smithi090.stdout:Expanding DB/WAL... 2021-05-12T12:17:19.414 INFO:tasks.workunit.client.0.smithi090.stdout:0 : expanding to 0x536870912 2021-05-12T12:17:19.448 INFO:tasks.workunit.client.0.smithi090.stdout:0 : size label updated to 536870912 2021-05-12T12:17:19.448 INFO:tasks.workunit.client.0.smithi090.stdout:1 : expanding to 0x1073741824 2021-05-12T12:17:19.473 INFO:tasks.workunit.client.0.smithi090.stdout:1 : size label updated to 1073741824 2021-05-12T12:17:19.473 INFO:tasks.workunit.client.0.smithi090.stdout:2 : expanding from 0x80000000 to 0xfff00000 2021-05-12T12:17:19.657 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:19.655+0000 7f90e3e53400 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T12:17:19.658 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:19.655+0000 7f90e3e53400 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T12:17:24.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:17:26.351 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:26.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:17:28.140 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:28.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:17:29.927 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:29.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:31.716 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:31.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2021-05-12T12:17:31.731 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:33.504 INFO:tasks.workunit.client.0.smithi090.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2021-05-12T12:17:33.504 INFO:tasks.workunit.client.0.smithi090.stdout:1 : device size 0x3fffe000 : using 0x1000000(16 MiB) 2021-05-12T12:17:33.505 INFO:tasks.workunit.client.0.smithi090.stdout:2 : device size 0x100000000 : using 0x4cf000(4.8 MiB) 2021-05-12T12:17:33.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2021-05-12T12:17:33.518 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:35.036 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T12:17:35.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:17:36.321 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:36.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T12:17:36.336 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:38.110 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:17:38.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:17:39.395 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:39.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2021-05-12T12:17:39.410 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:40.930 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T12:17:40.930 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T12:17:40.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:17:41.720 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:41.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2021-05-12T12:17:41.736 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:41.741 INFO:tasks.workunit.client.0.smithi090.stderr:Migrate to WAL device isn't supported. 2021-05-12T12:17:41.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T12:17:41.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:43.527 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:43.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T12:17:43.543 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:45.063 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T12:17:45.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:46.348 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:46.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2021-05-12T12:17:46.365 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:17:47.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:17:48.923 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:17:48.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:17:48.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:17:48.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:17:48.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:17:48.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:17:48.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:17:48.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:17:48.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:17:48.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:17:48.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:17:48.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:17:48.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:17:48.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:17:48.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:17:48.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:17:48.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:17:48.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:17:48.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:17:48.933 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:17:48.933 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:17:48.933 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:17:48.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:17:48.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:17:48.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:17:48.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:17:48.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:17:48.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:17:48.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:17:48.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:48.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:17:48.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:17:48.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:17:48.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:17:48.938 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:17:48.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:48.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:17:48.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:17:48.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:17:48.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:17:48.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:17:48.953 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:48.949+0000 7f2c5f8480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:48.954 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:48.951+0000 7f2c5f8480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:48.956 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:48.952+0000 7f2c5f8480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:49.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:17:49.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:17:49.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:17:49.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:17:49.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:17:49.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:49.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:17:49.303 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:17:49.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:49.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:49.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:49.721 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:49.718+0000 7f2c5f8480c0 -1 Falling back to public interface 2021-05-12T12:17:50.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:50.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:50.659 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:17:50.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:17:50.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:50.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:51.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:52.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:52.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:52.026 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:17:52.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:17:52.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:52.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:52.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:53.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:53.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:53.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:17:53.385 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:17:53.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:53.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:53.623+0000 7f2c5f8480c0 -1 osd.0 27 log_to_monitors {default=true} 2021-05-12T12:17:53.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:54.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:54.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:54.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:17:54.810 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:17:54.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:54.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:17:55.167 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/115881,v1:127.0.0.1:6803/115881] [v2:127.0.0.1:6804/115881,v1:127.0.0.1:6805/115881] exists,up 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:17:55.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:17:55.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:17:55.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:17:55.169 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:17:55.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=115881 2021-05-12T12:17:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:17:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:17:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:17:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:17:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:17:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:17:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:17:55.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:17:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:17:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:17:55.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:17:55.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:17:55.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:17:55.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:17:55.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:17:55.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:17:55.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:17:55.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:17:55.177 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:17:55.177 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:17:55.178 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:17:55.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:17:55.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:17:55.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:17:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:17:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:17:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:17:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:17:55.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:55.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:17:55.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:17:55.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:17:55.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:17:55.181 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:17:55.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:17:55.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:17:55.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:17:55.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:17:55.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:17:55.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:17:55.191 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:55.187+0000 7f13234010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:55.192 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:55.189+0000 7f13234010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:55.194 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:55.190+0000 7f13234010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:17:55.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:17:55.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:17:55.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:17:55.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:17:55.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:17:55.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:55.542 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:17:55.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:17:55.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:55.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:55.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:55.960 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:55.957+0000 7f13234010c0 -1 Falling back to public interface 2021-05-12T12:17:56.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:56.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:56.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:17:56.901 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:17:56.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:56.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:57.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:58.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:58.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:58.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:17:58.280 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:17:58.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:58.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:58.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:17:59.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:17:59.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:17:59.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:17:59.655 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:17:59.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:17:59.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:17:59.735 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:17:59.732+0000 7f13234010c0 -1 osd.1 29 log_to_monitors {default=true} 2021-05-12T12:18:00.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:01.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:01.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:01.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:18:01.035 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:18:01.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:01.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:18:01.413 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6810/116876,v1:127.0.0.1:6811/116876] [v2:127.0.0.1:6812/116876,v1:127.0.0.1:6813/116876] exists,up 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:18:01.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:18:01.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:18:01.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:18:01.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:18:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=116876 2021-05-12T12:18:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:18:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:18:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:18:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:18:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:18:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:18:01.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:18:01.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:18:01.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:18:01.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:18:01.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:18:01.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:18:01.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:18:01.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:18:01.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:18:01.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:18:01.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:18:01.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:18:01.424 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:18:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:18:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:18:01.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:18:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:18:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:18:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:18:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:18:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:18:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:18:01.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:01.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:18:01.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:18:01.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:18:01.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:18:01.428 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:18:01.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:01.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:18:01.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:18:01.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:18:01.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:18:01.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:18:01.438 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:01.435+0000 7fa6c25d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:01.440 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:01.437+0000 7fa6c25d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:01.441 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:01.438+0000 7fa6c25d00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:01.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:18:01.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:18:01.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:18:01.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:18:01.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:18:01.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:01.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:18:01.789 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:18:01.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:01.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:01.957 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:01.954+0000 7fa6c25d00c0 -1 Falling back to public interface 2021-05-12T12:18:02.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:03.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:03.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:03.172 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:18:03.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:18:03.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:03.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:03.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:04.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:04.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:04.538 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:18:04.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:18:04.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:04.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:04.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:05.286 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:05.283+0000 7fa6c25d00c0 -1 osd.2 33 log_to_monitors {default=true} 2021-05-12T12:18:05.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:05.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:05.926 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:18:05.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:18:05.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:05.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:18:06.634 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 34 last_clean_interval [14,33) [v2:127.0.0.1:6818/117863,v1:127.0.0.1:6819/117863] [v2:127.0.0.1:6820/117863,v1:127.0.0.1:6821/117863] exists,up d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:18:06.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:18:06.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:18:06.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:18:06.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:18:06.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=117863 2021-05-12T12:18:06.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:18:06.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:18:06.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:18:06.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:18:06.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:18:06.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:18:06.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:18:06.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:18:06.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:18:06.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:18:06.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:18:06.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:18:06.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:18:06.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:18:06.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:18:06.643 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:18:06.643 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:18:06.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:18:06.644 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:18:06.644 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:18:06.644 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:18:06.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:18:06.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:18:06.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:18:06.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:18:06.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:18:06.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:18:06.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:18:06.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:06.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:18:06.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:18:06.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:18:06.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:18:06.647 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:18:06.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:18:06.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:18:06.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:18:06.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:18:06.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:18:06.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:18:06.671 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:06.669+0000 7f5506d420c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:06.674 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:06.671+0000 7f5506d420c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:06.676 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:06.673+0000 7f5506d420c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:18:07.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:18:07.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:18:07.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:18:07.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:18:07.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:18:07.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:07.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:18:07.261 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:18:07.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:07.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:07.441 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:07.438+0000 7f5506d420c0 -1 Falling back to public interface 2021-05-12T12:18:07.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:08.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:08.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:08.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:18:08.632 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:18:08.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:08.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:08.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:09.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:09.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:09.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:18:09.993 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:18:09.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:09.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:10.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:11.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:11.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:11.351 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:18:11.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:18:11.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:11.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:11.451 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:18:11.448+0000 7f5506d420c0 -1 osd.3 35 log_to_monitors {default=true} 2021-05-12T12:18:11.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:18:12.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:18:12.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:18:12.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:18:12.767 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:18:12.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:18:12.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:18:13.336 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6826/118814,v1:127.0.0.1:6827/118814] [v2:127.0.0.1:6828/118814,v1:127.0.0.1:6829/118814] exists,up d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:18:13.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:18:13.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:18:13.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:18:13.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:18:13.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=118814 2021-05-12T12:18:13.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2021-05-12T12:18:13.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:18:13.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:18:13.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:18:13.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:18:13.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:18:13.341 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:18:13.341 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:18:13.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:18:13.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:18:13.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:18:13.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:18:13.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:18:13.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:18:13.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:18:13.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:18:13.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:18:13.793 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:18:13.793 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:18:13.793 INFO:tasks.workunit.client.0.smithi090.stderr:3' 2021-05-12T12:18:13.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:18:13.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:13.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:18:13.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=163208757255 2021-05-12T12:18:13.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 163208757255 2021-05-12T12:18:13.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757255' 2021-05-12T12:18:13.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:13.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:18:14.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659141 2021-05-12T12:18:14.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659141 2021-05-12T12:18:14.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2021-05-12T12:18:14.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:14.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:18:14.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=193273528324 2021-05-12T12:18:14.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 193273528324 2021-05-12T12:18:14.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324' 2021-05-12T12:18:14.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:18:14.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:18:14.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=210453397507 2021-05-12T12:18:14.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 210453397507 2021-05-12T12:18:14.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324 3-210453397507' 2021-05-12T12:18:14.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:14.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-163208757255 2021-05-12T12:18:14.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:14.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:18:14.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-163208757255 2021-05-12T12:18:14.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:14.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=163208757255 2021-05-12T12:18:14.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2021-05-12T12:18:14.326 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 163208757255 2021-05-12T12:18:14.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:18:14.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 163208757252 -lt 163208757255 2021-05-12T12:18:14.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:18:15.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:18:15.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:18:16.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 163208757255 -lt 163208757255 2021-05-12T12:18:16.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:16.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-176093659141 2021-05-12T12:18:16.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:16.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:18:16.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-176093659141 2021-05-12T12:18:16.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:16.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659141 2021-05-12T12:18:16.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2021-05-12T12:18:16.047 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 176093659141 2021-05-12T12:18:16.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:18:16.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659141 -lt 176093659141 2021-05-12T12:18:16.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:16.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-193273528324 2021-05-12T12:18:16.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:16.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:18:16.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-193273528324 2021-05-12T12:18:16.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:16.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=193273528324 2021-05-12T12:18:16.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2021-05-12T12:18:16.429 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 193273528324 2021-05-12T12:18:16.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:18:16.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 193273528325 -lt 193273528324 2021-05-12T12:18:16.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:18:16.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-210453397507 2021-05-12T12:18:16.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:18:16.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:18:16.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-210453397507 2021-05-12T12:18:16.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:18:16.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=210453397507 2021-05-12T12:18:16.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2021-05-12T12:18:16.789 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 210453397507 2021-05-12T12:18:16.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:18:17.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 210453397507 -lt 210453397507 2021-05-12T12:18:17.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:18:17.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:18:17.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:18:17.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:18:17.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:18:17.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:18:17.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:18:17.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:18:17.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:18:17.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:18:17.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:18:17.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:18:17.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:18:17.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:18:17.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:18:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:18:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:18:18.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:18:18.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:18:18.486 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:18:18.486 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:18:18.487 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_120233 2021-05-12T12:18:18.487 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:18:18.487 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:18:19.486 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 33 17 0.0663982 0.0664062 0.23087 0.271984 2021-05-12T12:18:20.486 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 43 27 0.0527275 0.0390625 0.169768 0.254243 2021-05-12T12:18:21.486 INFO:tasks.workunit.client.0.smithi090.stdout: 3 16 43 27 0.0351516 0 - 0.254243 2021-05-12T12:18:22.487 INFO:tasks.workunit.client.0.smithi090.stdout: 4 16 43 27 0.0263636 0 - 0.254243 2021-05-12T12:18:23.487 INFO:tasks.workunit.client.0.smithi090.stdout: 5 16 61 45 0.0351514 0.0234375 0.222667 1.03155 2021-05-12T12:18:24.487 INFO:tasks.workunit.client.0.smithi090.stdout: 6 16 80 64 0.0416612 0.0742188 0.219551 0.804067 2021-05-12T12:18:25.487 INFO:tasks.workunit.client.0.smithi090.stdout: 7 16 116 100 0.0557962 0.140625 0.279387 1.10387 2021-05-12T12:18:26.487 INFO:tasks.workunit.client.0.smithi090.stdout: 8 16 152 136 0.0663974 0.140625 0.395856 0.902788 2021-05-12T12:18:27.487 INFO:tasks.workunit.client.0.smithi090.stdout: 9 16 212 196 0.0850583 0.234375 0.321278 0.724508 2021-05-12T12:18:28.487 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16 248 232 0.0906131 0.140625 0.408308 0.669397 2021-05-12T12:18:29.488 INFO:tasks.workunit.client.0.smithi090.stdout: 11 16 308 292 0.10368 0.234375 0.290031 0.59413 2021-05-12T12:18:30.488 INFO:tasks.workunit.client.0.smithi090.stdout: 12 16 356 340 0.110662 0.1875 0.275599 0.555021 2021-05-12T12:18:31.488 INFO:tasks.workunit.client.0.smithi090.stdout: 13 16 408 392 0.117772 0.203125 0.426368 0.521896 2021-05-12T12:18:32.488 INFO:tasks.workunit.client.0.smithi090.stdout: 14 16 468 452 0.126099 0.234375 0.316944 0.494132 2021-05-12T12:18:33.488 INFO:tasks.workunit.client.0.smithi090.stdout: 15 16 516 500 0.130191 0.1875 0.333557 0.476714 2021-05-12T12:18:34.488 INFO:tasks.workunit.client.0.smithi090.stdout: 16 16 568 552 0.134748 0.203125 0.315982 0.46032 2021-05-12T12:18:35.488 INFO:tasks.workunit.client.0.smithi090.stdout: 17 16 616 600 0.137849 0.1875 0.337756 0.449913 2021-05-12T12:18:36.488 INFO:tasks.workunit.client.0.smithi090.stdout: 18 16 664 648 0.140606 0.1875 0.427091 0.437828 2021-05-12T12:18:37.489 INFO:tasks.workunit.client.0.smithi090.stdout: 19 16 712 696 0.143073 0.1875 0.353022 0.431629 2021-05-12T12:18:38.489 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:18:38.486841+0000 min lat: 0.116614 max lat: 6.67087 avg lat: 0.425123 2021-05-12T12:18:38.489 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:18:38.489 INFO:tasks.workunit.client.0.smithi090.stdout: 20 16 760 744 0.145294 0.1875 0.388575 0.425123 2021-05-12T12:18:39.489 INFO:tasks.workunit.client.0.smithi090.stdout: 21 16 808 792 0.147302 0.1875 0.309603 0.419657 2021-05-12T12:18:40.489 INFO:tasks.workunit.client.0.smithi090.stdout: 22 16 868 852 0.151258 0.234375 0.341513 0.411116 2021-05-12T12:18:41.489 INFO:tasks.workunit.client.0.smithi090.stdout: 23 16 916 900 0.152833 0.1875 0.285392 0.405564 2021-05-12T12:18:42.489 INFO:tasks.workunit.client.0.smithi090.stdout: 24 16 968 952 0.154927 0.203125 0.274586 0.400571 2021-05-12T12:18:43.489 INFO:tasks.workunit.client.0.smithi090.stdout: 25 16 1016 1000 0.156229 0.1875 0.328383 0.397283 2021-05-12T12:18:44.489 INFO:tasks.workunit.client.0.smithi090.stdout: 26 16 1064 1048 0.157431 0.1875 0.325496 0.394032 2021-05-12T12:18:45.489 INFO:tasks.workunit.client.0.smithi090.stdout: 27 16 1112 1096 0.158544 0.1875 0.366485 0.39086 2021-05-12T12:18:46.490 INFO:tasks.workunit.client.0.smithi090.stdout: 28 16 1160 1144 0.159578 0.1875 0.333379 0.388448 2021-05-12T12:18:47.490 INFO:tasks.workunit.client.0.smithi090.stdout: 29 16 1220 1204 0.162156 0.234375 0.28298 0.383768 2021-05-12T12:18:48.490 INFO:tasks.workunit.client.0.smithi090.stdout: 30 16 1272 1256 0.163521 0.203125 0.266676 0.379796 2021-05-12T12:18:49.490 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 30.232 2021-05-12T12:18:49.491 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 1272 2021-05-12T12:18:49.491 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:18:49.491 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:18:49.491 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.164354 2021-05-12T12:18:49.492 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.0708992 2021-05-12T12:18:49.492 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.234375 2021-05-12T12:18:49.492 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:18:49.492 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 42 2021-05-12T12:18:49.492 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 18.1502 2021-05-12T12:18:49.493 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 60 2021-05-12T12:18:49.493 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 0 2021-05-12T12:18:49.493 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.378769 2021-05-12T12:18:49.493 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 0.548718 2021-05-12T12:18:49.494 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 6.67087 2021-05-12T12:18:49.494 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.116614 2021-05-12T12:18:49.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 115881 2021-05-12T12:18:49.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2021-05-12T12:18:50.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 115881 2021-05-12T12:18:50.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (115881) - No such process 2021-05-12T12:18:50.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2021-05-12T12:18:51.044 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:18:51.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 116876 2021-05-12T12:18:51.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2021-05-12T12:18:52.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 116876 2021-05-12T12:18:52.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (116876) - No such process 2021-05-12T12:18:52.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2021-05-12T12:18:52.753 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:18:52.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 117863 2021-05-12T12:18:52.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2021-05-12T12:18:53.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 117863 2021-05-12T12:18:53.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (117863) - No such process 2021-05-12T12:18:53.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2021-05-12T12:18:54.147 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:18:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 118814 2021-05-12T12:18:54.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2021-05-12T12:18:55.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 118814 2021-05-12T12:18:55.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (118814) - No such process 2021-05-12T12:18:55.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2021-05-12T12:18:55.636 INFO:tasks.workunit.client.0.smithi090.stderr:osd.3 is already down. 2021-05-12T12:18:55.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:18:56.926 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:18:56.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2021-05-12T12:18:57.186 INFO:tasks.workunit.client.0.smithi090.stderr:512+0 records in 2021-05-12T12:18:57.186 INFO:tasks.workunit.client.0.smithi090.stderr:512+0 records out 2021-05-12T12:18:57.186 INFO:tasks.workunit.client.0.smithi090.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.25623 s, 2.1 GB/s 2021-05-12T12:18:57.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2021-05-12T12:18:57.196 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:02.717 INFO:tasks.workunit.client.0.smithi090.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2021-05-12T12:19:02.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:19:04.503 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:04.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:19:05.789 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:05.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2021-05-12T12:19:06.278 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:19:06.278 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:19:06.278 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.4847 s, 2.2 GB/s 2021-05-12T12:19:06.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2021-05-12T12:19:06.288 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:16.561 INFO:tasks.workunit.client.0.smithi090.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2021-05-12T12:19:16.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2021-05-12T12:19:16.575 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:18.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:19:20.132 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:20.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:19:20.920 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:20.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2021-05-12T12:19:20.936 INFO:tasks.workunit.client.0.smithi090.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:20.936 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:21.706 INFO:tasks.workunit.client.0.smithi090.stdout:DB device added 2021-05-12T12:19:21.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2021-05-12T12:19:21.720 INFO:tasks.workunit.client.0.smithi090.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:21.721 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:22.989 INFO:tasks.workunit.client.0.smithi090.stdout:WAL device added 2021-05-12T12:19:22.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2021-05-12T12:19:23.004 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:24.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:19:26.560 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:26.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:19:27.846 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:27.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2021-05-12T12:19:27.860 INFO:tasks.workunit.client.0.smithi090.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T12:19:27.860 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:29.130 INFO:tasks.workunit.client.0.smithi090.stdout:WAL device added 2021-05-12T12:19:29.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T12:19:29.144 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:19:30.917 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2021-05-12T12:19:30.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:19:32.203 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:19:32.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:19:32.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:32.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:32.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:19:32.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:32.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:19:32.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:32.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:32.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:32.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:32.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:19:32.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:19:32.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:32.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:32.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:32.213 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:32.213 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:32.214 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:19:32.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:32.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:32.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:32.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:32.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:32.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:32.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:32.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:32.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:32.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:32.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:19:32.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:19:32.217 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:19:32.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:32.218 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:19:32.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:19:32.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:32.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:32.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:32.232 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:32.228+0000 7f578415d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:32.234 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:32.231+0000 7f578415d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:32.239 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:32.232+0000 7f578415d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:32.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:19:32.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:32.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:19:32.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:32.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:32.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:32.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:32.579 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:19:32.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:32.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:33.248+0000 7f578415d0c0 -1 Falling back to public interface 2021-05-12T12:19:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:33.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:33.939 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:19:33.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:33.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:34.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:35.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:35.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:35.298 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:19:35.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:35.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:35.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:35.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:36.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:36.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:36.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:36.661 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:19:36.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:36.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:37.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:37.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:37.796+0000 7f578415d0c0 -1 osd.0 51 log_to_monitors {default=true} 2021-05-12T12:19:38.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:38.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:38.024 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:19:38.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:38.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:38.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:19:38.379 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 62 up_thru 62 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6802/126430,v1:127.0.0.1:6803/126430] [v2:127.0.0.1:6804/126430,v1:127.0.0.1:6805/126430] exists,up 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:19:38.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:38.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:38.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:38.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:19:38.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=126430 2021-05-12T12:19:38.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:19:38.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:38.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:38.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:19:38.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:38.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:19:38.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:38.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:38.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:38.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:38.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:19:38.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:19:38.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:38.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:38.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:38.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:38.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:38.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:38.389 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:38.389 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:38.390 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:19:38.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:38.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:38.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:38.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:38.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:38.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:38.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:38.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:38.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:38.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:38.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:19:38.393 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:19:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:19:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:38.394 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:19:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:19:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:38.403 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:38.400+0000 7eff3cc560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:38.416 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:38.414+0000 7eff3cc560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:38.418 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:38.415+0000 7eff3cc560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:38.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:19:38.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:38.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:19:38.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:38.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:38.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:38.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:38.744 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:19:38.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:38.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:39.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:39.435 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:39.432+0000 7eff3cc560c0 -1 Falling back to public interface 2021-05-12T12:19:40.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:40.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:40.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:40.098 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:19:40.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:40.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:40.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:41.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:41.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:41.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:41.490 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:19:41.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:41.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:41.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:42.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:42.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:42.850 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:19:42.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:42.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:42.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:43.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:43.852 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:43.848+0000 7eff3cc560c0 -1 osd.1 54 log_to_monitors {default=true} 2021-05-12T12:19:44.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:44.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:44.211 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:19:44.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:44.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:44.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:19:44.588 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/127427,v1:127.0.0.1:6811/127427] [v2:127.0.0.1:6812/127427,v1:127.0.0.1:6813/127427] exists,up 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:19:44.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:44.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:44.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:44.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:19:44.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=127427 2021-05-12T12:19:44.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:19:44.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:44.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:44.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:19:44.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:44.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:19:44.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:44.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:44.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:19:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:19:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:44.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:44.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:44.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:44.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:44.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:19:44.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:19:44.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:19:44.601 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:19:44.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:44.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:19:44.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:19:44.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:44.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:44.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:44.612 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:44.608+0000 7f53968a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:44.613 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:44.610+0000 7f53968a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:44.615 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:44.611+0000 7f53968a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:44.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:19:44.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:44.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:19:44.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:44.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:44.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:44.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:44.981 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:19:44.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:44.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:45.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:45.631 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:45.627+0000 7f53968a80c0 -1 Falling back to public interface 2021-05-12T12:19:46.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:46.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:46.331 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:19:46.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:46.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:46.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:46.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:47.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:47.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:47.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:47.692 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:19:47.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:47.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:48.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:49.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:49.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:49.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:49.054 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:19:49.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:49.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:49.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:50.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:50.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:50.413 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:19:50.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:50.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:50.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:50.568 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:50.564+0000 7f53968a80c0 -1 osd.2 57 log_to_monitors {default=true} 2021-05-12T12:19:50.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:51.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:51.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:51.784 INFO:tasks.workunit.client.0.smithi090.stdout:5 2021-05-12T12:19:51.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T12:19:51.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:51.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:19:52.447 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 68 up_thru 68 down_at 58 last_clean_interval [45,57) [v2:127.0.0.1:6818/128424,v1:127.0.0.1:6819/128424] [v2:127.0.0.1:6820/128424,v1:127.0.0.1:6821/128424] exists,up d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:19:52.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:52.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:52.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:52.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:19:52.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=128424 2021-05-12T12:19:52.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:19:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:19:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:19:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:19:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:19:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:19:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:19:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:19:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:19:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:19:52.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:19:52.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:19:52.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:19:52.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:19:52.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:19:52.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:19:52.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:19:52.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:19:52.457 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:19:52.457 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:19:52.457 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:19:52.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:52.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:19:52.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:19:52.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:19:52.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:19:52.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:19:52.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:19:52.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:52.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:19:52.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:19:52.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:19:52.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:19:52.460 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:19:52.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:19:52.462 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:19:52.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:19:52.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:19:52.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:19:52.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:19:52.487 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:52.484+0000 7f74d01b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:52.490 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:52.486+0000 7f74d01b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:52.491 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:52.487+0000 7f74d01b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:19:52.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:19:52.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:19:52.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:19:52.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:19:52.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:19:52.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:52.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:19:52.852 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:19:52.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:52.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:53.258 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:53.254+0000 7f74d01b70c0 -1 Falling back to public interface 2021-05-12T12:19:53.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:54.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:54.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:19:54.282 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:19:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:54.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:54.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:55.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:55.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:55.642 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:19:55.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:19:55.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:55.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:56.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:57.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:57.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:57.065 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:19:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:19:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:57.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:19:57.665 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:19:57.661+0000 7f74d01b70c0 -1 osd.3 59 log_to_monitors {default=true} 2021-05-12T12:19:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:19:58.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:19:58.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:19:58.436 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:19:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:19:58.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:19:58.879 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 60 last_clean_interval [49,59) [v2:127.0.0.1:6826/129447,v1:127.0.0.1:6827/129447] [v2:127.0.0.1:6828/129447,v1:127.0.0.1:6829/129447] exists,up d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:19:58.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:19:58.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:19:58.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:19:58.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:19:58.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=129447 2021-05-12T12:19:58.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:19:58.907 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:19:58.907 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:19:58.908 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_130427 2021-05-12T12:19:58.908 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:19:58.908 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:19:59.907 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 25 9 0.0351536 0.0351562 0.323648 0.281803 2021-05-12T12:20:00.907 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 42 26 0.050776 0.0664062 0.313487 0.399701 2021-05-12T12:20:01.908 INFO:tasks.workunit.client.0.smithi090.stdout: 3 16 50 34 0.044266 0.03125 0.167839 0.360686 2021-05-12T12:20:02.908 INFO:tasks.workunit.client.0.smithi090.stdout: 4 16 50 34 0.0331993 0 - 0.360686 2021-05-12T12:20:03.908 INFO:tasks.workunit.client.0.smithi090.stdout: 5 16 50 34 0.0265593 0 - 0.360686 2021-05-12T12:20:04.908 INFO:tasks.workunit.client.0.smithi090.stdout: 6 16 50 34 0.0221328 0 - 0.360686 2021-05-12T12:20:05.908 INFO:tasks.workunit.client.0.smithi090.stdout: 7 16 50 34 0.0189709 0 - 0.360686 2021-05-12T12:20:06.908 INFO:tasks.workunit.client.0.smithi090.stdout: 8 16 50 34 0.0165995 0 - 0.360686 2021-05-12T12:20:07.908 INFO:tasks.workunit.client.0.smithi090.stdout: 9 16 50 34 0.0147551 0 - 0.360686 2021-05-12T12:20:08.911 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16 86 70 0.0273404 0.0200893 0.209221 2.25663 2021-05-12T12:20:09.908 INFO:tasks.workunit.client.0.smithi090.stdout: 11 16 131 115 0.0408332 0.175781 0.310583 1.50351 2021-05-12T12:20:10.909 INFO:tasks.workunit.client.0.smithi090.stdout: 12 16 178 162 0.0527281 0.183594 0.47244 1.16889 2021-05-12T12:20:11.909 INFO:tasks.workunit.client.0.smithi090.stdout: 13 16 223 207 0.0621923 0.175781 0.435633 0.993627 2021-05-12T12:20:12.909 INFO:tasks.workunit.client.0.smithi090.stdout: 14 16 271 255 0.0711412 0.1875 0.308648 0.867123 2021-05-12T12:20:13.909 INFO:tasks.workunit.client.0.smithi090.stdout: 15 16 322 306 0.0796783 0.199219 0.182273 0.776887 2021-05-12T12:20:14.909 INFO:tasks.workunit.client.0.smithi090.stdout: 16 16 372 356 0.086904 0.195312 0.318722 0.711485 2021-05-12T12:20:15.909 INFO:tasks.workunit.client.0.smithi090.stdout: 17 16 424 408 0.093739 0.203125 0.193742 0.661463 2021-05-12T12:20:16.909 INFO:tasks.workunit.client.0.smithi090.stdout: 18 16 461 445 0.0965597 0.144531 0.406482 0.635289 2021-05-12T12:20:17.910 INFO:tasks.workunit.client.0.smithi090.stdout: 19 16 510 494 0.101551 0.191406 0.17653 0.609478 2021-05-12T12:20:18.910 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:20:18.907719+0000 min lat: 0.151873 max lat: 9.76583 avg lat: 0.581694 2021-05-12T12:20:18.910 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:20:18.910 INFO:tasks.workunit.client.0.smithi090.stdout: 20 16 563 547 0.106824 0.207031 0.358629 0.581694 2021-05-12T12:20:19.910 INFO:tasks.workunit.client.0.smithi090.stdout: 21 16 613 597 0.111036 0.195312 0.283776 0.558906 2021-05-12T12:20:20.910 INFO:tasks.workunit.client.0.smithi090.stdout: 22 16 664 648 0.115043 0.199219 0.219581 0.538755 2021-05-12T12:20:21.910 INFO:tasks.workunit.client.0.smithi090.stdout: 23 16 712 696 0.118192 0.1875 0.31216 0.523683 2021-05-12T12:20:22.910 INFO:tasks.workunit.client.0.smithi090.stdout: 24 16 758 742 0.120753 0.179688 0.35055 0.514505 2021-05-12T12:20:23.910 INFO:tasks.workunit.client.0.smithi090.stdout: 25 16 807 791 0.123578 0.191406 0.202923 0.502912 2021-05-12T12:20:24.910 INFO:tasks.workunit.client.0.smithi090.stdout: 26 16 855 839 0.126036 0.1875 0.358841 0.493826 2021-05-12T12:20:25.911 INFO:tasks.workunit.client.0.smithi090.stdout: 27 16 904 888 0.128456 0.191406 0.317214 0.483295 2021-05-12T12:20:26.911 INFO:tasks.workunit.client.0.smithi090.stdout: 28 16 959 943 0.131541 0.214844 0.249618 0.472506 2021-05-12T12:20:27.911 INFO:tasks.workunit.client.0.smithi090.stdout: 29 16 1003 987 0.13293 0.171875 0.465271 0.467268 2021-05-12T12:20:28.911 INFO:tasks.workunit.client.0.smithi090.stdout: 30 16 1056 1040 0.1354 0.207031 0.263718 0.458706 2021-05-12T12:20:29.911 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 30.1396 2021-05-12T12:20:29.912 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 1056 2021-05-12T12:20:29.912 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:20:29.912 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:20:29.912 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.136863 2021-05-12T12:20:29.912 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.0853933 2021-05-12T12:20:29.913 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.214844 2021-05-12T12:20:29.913 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:20:29.913 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 35 2021-05-12T12:20:29.913 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 21.8671 2021-05-12T12:20:29.913 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 55 2021-05-12T12:20:29.914 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 0 2021-05-12T12:20:29.914 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.455916 2021-05-12T12:20:29.914 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 1.03145 2021-05-12T12:20:29.914 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 9.76583 2021-05-12T12:20:29.915 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.151873 2021-05-12T12:20:30.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 126430 2021-05-12T12:20:30.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2021-05-12T12:20:31.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 126430 2021-05-12T12:20:31.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (126430) - No such process 2021-05-12T12:20:31.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2021-05-12T12:20:31.504 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:20:31.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 127427 2021-05-12T12:20:31.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2021-05-12T12:20:32.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 127427 2021-05-12T12:20:32.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (127427) - No such process 2021-05-12T12:20:32.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2021-05-12T12:20:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:osd.1 is already down. 2021-05-12T12:20:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 128424 2021-05-12T12:20:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2021-05-12T12:20:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 128424 2021-05-12T12:20:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (128424) - No such process 2021-05-12T12:20:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2021-05-12T12:20:34.319 INFO:tasks.workunit.client.0.smithi090.stderr:osd.2 is already down. 2021-05-12T12:20:34.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 129447 2021-05-12T12:20:34.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2021-05-12T12:20:35.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 129447 2021-05-12T12:20:35.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (129447) - No such process 2021-05-12T12:20:35.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2021-05-12T12:20:35.830 INFO:tasks.workunit.client.0.smithi090.stderr:osd.3 is already down. 2021-05-12T12:20:35.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:37.620 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:20:37.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2021-05-12T12:20:38.103 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:20:38.103 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:20:38.104 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.479376 s, 2.2 GB/s 2021-05-12T12:20:38.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2021-05-12T12:20:38.113 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:20:49.136 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2021-05-12T12:20:49.136 INFO:tasks.workunit.client.0.smithi090.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2021-05-12T12:20:49.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:50.923 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:20:50.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2021-05-12T12:20:51.178 INFO:tasks.workunit.client.0.smithi090.stderr:512+0 records in 2021-05-12T12:20:51.178 INFO:tasks.workunit.client.0.smithi090.stderr:512+0 records out 2021-05-12T12:20:51.178 INFO:tasks.workunit.client.0.smithi090.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.250981 s, 2.1 GB/s 2021-05-12T12:20:51.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2021-05-12T12:20:51.188 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:20:57.208 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T12:20:57.209 INFO:tasks.workunit.client.0.smithi090.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2021-05-12T12:20:57.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2021-05-12T12:20:57.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T12:20:59.048 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:20:59.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:21:00.835 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:00.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2021-05-12T12:21:01.326 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:21:01.327 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:21:01.327 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.487227 s, 2.2 GB/s 2021-05-12T12:21:01.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2021-05-12T12:21:01.336 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:12.359 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:21:12.359 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2021-05-12T12:21:12.360 INFO:tasks.workunit.client.0.smithi090.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2021-05-12T12:21:12.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2021-05-12T12:21:12.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:21:13.735 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:13.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T12:21:13.751 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:15.026 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T12:21:15.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2021-05-12T12:21:15.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T12:21:15.903 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:15.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2021-05-12T12:21:17.686 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:17.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T12:21:18.170 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:21:18.170 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:21:18.170 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.479746 s, 2.2 GB/s 2021-05-12T12:21:18.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T12:21:18.179 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:18.184 INFO:tasks.workunit.client.0.smithi090.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2021-05-12T12:21:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T12:21:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:21:19.965 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:19.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T12:21:29.246 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:21:29.247 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:21:29.247 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.14787 s, 117 MB/s 2021-05-12T12:21:29.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T12:21:29.259 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T12:21:31.530 INFO:tasks.workunit.client.0.smithi090.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2021-05-12T12:21:31.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T12:21:32.817 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:32.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2021-05-12T12:21:33.300 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records in 2021-05-12T12:21:33.301 INFO:tasks.workunit.client.0.smithi090.stderr:1024+0 records out 2021-05-12T12:21:33.301 INFO:tasks.workunit.client.0.smithi090.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.479769 s, 2.2 GB/s 2021-05-12T12:21:33.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2021-05-12T12:21:33.310 INFO:tasks.workunit.client.0.smithi090.stdout:inferring bluefs devices from bluestore path 2021-05-12T12:21:43.334 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2021-05-12T12:21:43.335 INFO:tasks.workunit.client.0.smithi090.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T12:21:43.335 INFO:tasks.workunit.client.0.smithi090.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2021-05-12T12:21:43.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T12:21:44.624 INFO:tasks.workunit.client.0.smithi090.stdout:fsck success 2021-05-12T12:21:44.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T12:21:44.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:44.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:44.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:21:44.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:44.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T12:21:44.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:44.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:44.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:44.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:44.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T12:21:44.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T12:21:44.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:44.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:44.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:44.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:44.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:44.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:44.633 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:44.633 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:44.633 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:21:44.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:44.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:44.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:44.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:44.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:44.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:44.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:44.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:44.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:44.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:44.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T12:21:44.638 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:21:44.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:21:44.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:44.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T12:21:44.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:21:44.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:44.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:44.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:44.652 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:44.648+0000 7fa1da7db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:44.654 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:44.651+0000 7fa1da7db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:44.655 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:44.652+0000 7fa1da7db0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:44.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:21:45.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:45.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:21:45.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:45.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:45.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:45.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:45.001 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:21:45.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:45.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:45.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:45.672 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:45.669+0000 7fa1da7db0c0 -1 Falling back to public interface 2021-05-12T12:21:46.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:46.353 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:21:46.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:46.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:46.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:46.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:47.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:47.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:47.710 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:21:47.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:47.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:47.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:48.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:49.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:49.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:49.068 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:21:49.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:49.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:49.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:49.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:50.157 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:50.153+0000 7fa1da7db0c0 -1 osd.0 74 log_to_monitors {default=true} 2021-05-12T12:21:50.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:50.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:50.429 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:21:50.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T12:21:50.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:50.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:21:50.835 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 75 last_clean_interval [62,74) [v2:127.0.0.1:6802/136661,v1:127.0.0.1:6803/136661] [v2:127.0.0.1:6804/136661,v1:127.0.0.1:6805/136661] exists,up 64254159-a6cd-4e09-9277-b2d729d7b59e 2021-05-12T12:21:50.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:50.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:50.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:50.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T12:21:50.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=136661 2021-05-12T12:21:50.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T12:21:50.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:50.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:50.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:21:50.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:50.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T12:21:50.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:50.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:50.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:50.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:50.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T12:21:50.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T12:21:50.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:50.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:50.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:50.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:50.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:50.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:50.844 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:50.845 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:50.845 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:21:50.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:50.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:50.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:50.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:50.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:50.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:50.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:50.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:50.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:50.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:50.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T12:21:50.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:21:50.848 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:21:50.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:50.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T12:21:50.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:21:50.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:50.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:50.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:50.859 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:50.855+0000 7fa906ff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:50.880 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:50.877+0000 7fa906ff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:50.881 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:50.878+0000 7fa906ff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:21:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:21:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:51.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:51.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:51.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:51.201 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:21:51.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:51.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:51.401 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:51.398+0000 7fa906ff10c0 -1 Falling back to public interface 2021-05-12T12:21:51.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:52.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:52.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:52.555 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:21:52.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:52.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:52.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:52.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:53.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:53.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:53.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:53.918 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:21:53.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:53.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:54.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:54.793 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:54.789+0000 7fa906ff10c0 -1 osd.1 77 log_to_monitors {default=true} 2021-05-12T12:21:55.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:55.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:55.274 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:21:55.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:21:55.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:55.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:21:55.686 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 87 up_thru 75 down_at 78 last_clean_interval [65,77) [v2:127.0.0.1:6810/137658,v1:127.0.0.1:6811/137658] [v2:127.0.0.1:6812/137658,v1:127.0.0.1:6813/137658] exists,up 5e29472e-2a43-441d-ac2f-cced9184cbe1 2021-05-12T12:21:55.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:21:55.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:21:55.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:21:55.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T12:21:55.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=137658 2021-05-12T12:21:55.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T12:21:55.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:21:55.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:21:55.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:21:55.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:21:55.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T12:21:55.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:21:55.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:21:55.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:21:55.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:21:55.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T12:21:55.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T12:21:55.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:21:55.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:21:55.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:21:55.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:21:55.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:21:55.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:21:55.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:21:55.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:21:55.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:21:55.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:55.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:21:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:21:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:21:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:21:55.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:21:55.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:21:55.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:55.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:21:55.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:21:55.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T12:21:55.699 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:21:55.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:21:55.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:21:55.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T12:21:55.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:21:55.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:21:55.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:21:55.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:21:55.710 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:55.706+0000 7fd24ecaf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:55.712 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:55.708+0000 7fd24ecaf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:55.713 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:55.710+0000 7fd24ecaf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:21:56.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:21:56.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:21:56.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:21:56.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:21:56.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:21:56.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:56.208 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:21:56.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:21:56.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:56.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:56.482 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:21:56.478+0000 7fd24ecaf0c0 -1 Falling back to public interface 2021-05-12T12:21:56.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:57.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:57.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:57.566 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:21:57.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:21:57.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:57.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:57.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:21:58.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:21:58.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:21:58.924 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:21:58.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:21:58.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:21:58.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:21:59.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:00.179 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:00.176+0000 7fd24ecaf0c0 -1 osd.2 79 log_to_monitors {default=true} 2021-05-12T12:22:00.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:00.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:00.310 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:22:00.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:22:00.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:00.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:22:00.826 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 91 up_thru 78 down_at 80 last_clean_interval [68,79) [v2:127.0.0.1:6818/138617,v1:127.0.0.1:6819/138617] [v2:127.0.0.1:6820/138617,v1:127.0.0.1:6821/138617] exists,up d54e3313-a346-4e73-be73-a214d6572386 2021-05-12T12:22:00.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:22:00.828 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:22:00.828 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:22:00.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T12:22:00.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=138617 2021-05-12T12:22:00.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T12:22:00.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:22:00.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:22:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T12:22:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:22:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T12:22:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T12:22:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T12:22:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T12:22:00.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:00.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:22:00.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T12:22:00.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:22:00.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:00.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:00.836 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:00.836 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:00.836 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:22:00.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:22:00.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' 2021-05-12T12:22:00.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:00.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T12:22:00.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T12:22:00.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:00.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:00.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:00.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:22:00.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:22:00.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T12:22:00.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T12:22:00.840 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:22:00.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=e7daf2d9-04c1-49dc-871c-73af7604c8a7 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94350/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:00.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T12:22:00.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T12:22:00.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:22:00.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:22:00.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:22:00.864 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:00.862+0000 7f8a595e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:00.867 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:00.864+0000 7f8a595e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:00.869 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:00.866+0000 7f8a595e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:01.636 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:01.633+0000 7f8a595e90c0 -1 Falling back to public interface 2021-05-12T12:22:01.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T12:22:01.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:22:01.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:22:01.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:22:01.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:22:01.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:01.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:22:01.648 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:22:01.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:01.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:22:02.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:03.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:03.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:03.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:22:03.225 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:22:03.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:03.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:22:03.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:04.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:04.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:04.588 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:22:04.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:22:04.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:04.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:22:04.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:05.498 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:05.495+0000 7f8a595e90c0 -1 osd.3 81 log_to_monitors {default=true} 2021-05-12T12:22:05.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:05.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:05.972 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:22:05.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:22:05.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:05.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:22:06.481 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 95 up_thru 95 down_at 82 last_clean_interval [72,81) [v2:127.0.0.1:6826/139580,v1:127.0.0.1:6827/139580] [v2:127.0.0.1:6828/139580,v1:127.0.0.1:6829/139580] exists,up d71baae8-2593-4a1c-95b4-3821273338a2 2021-05-12T12:22:06.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:22:06.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:22:06.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:22:06.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T12:22:06.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=139580 2021-05-12T12:22:06.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T12:22:06.507 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:22:06.508 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T12:22:06.508 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_140526 2021-05-12T12:22:06.508 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:22:06.508 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:22:07.508 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 29 13 0.0507766 0.0507812 0.866534 0.591451 2021-05-12T12:22:08.508 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 53 37 0.0722592 0.09375 0.0583857 0.297828 2021-05-12T12:22:09.508 INFO:tasks.workunit.client.0.smithi090.stdout: 3 16 55 39 0.0507761 0.0078125 0.0682141 0.286448 2021-05-12T12:22:10.508 INFO:tasks.workunit.client.0.smithi090.stdout: 4 16 55 39 0.0380819 0 - 0.286448 2021-05-12T12:22:11.508 INFO:tasks.workunit.client.0.smithi090.stdout: 5 16 55 39 0.0304654 0 - 0.286448 2021-05-12T12:22:12.508 INFO:tasks.workunit.client.0.smithi090.stdout: 6 16 55 39 0.0253876 0 - 0.286448 2021-05-12T12:22:13.508 INFO:tasks.workunit.client.0.smithi090.stdout: 7 16 55 39 0.0217607 0 - 0.286448 2021-05-12T12:22:14.508 INFO:tasks.workunit.client.0.smithi090.stdout: 8 16 55 39 0.0190407 0 - 0.286448 2021-05-12T12:22:15.509 INFO:tasks.workunit.client.0.smithi090.stdout: 9 16 55 39 0.0169251 0 - 0.286448 2021-05-12T12:22:16.509 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16 55 39 0.0152326 0 - 0.286448 2021-05-12T12:22:17.509 INFO:tasks.workunit.client.0.smithi090.stdout: 11 16 55 39 0.0138478 0 - 0.286448 2021-05-12T12:22:18.509 INFO:tasks.workunit.client.0.smithi090.stdout: 12 16 100 84 0.0273404 0.0195312 0.195178 1.71705 2021-05-12T12:22:19.509 INFO:tasks.workunit.client.0.smithi090.stdout: 13 16 135 119 0.0357528 0.136719 0.10504 1.26727 2021-05-12T12:22:20.509 INFO:tasks.workunit.client.0.smithi090.stdout: 14 16 188 172 0.0479851 0.207031 0.125569 1.28923 2021-05-12T12:22:21.509 INFO:tasks.workunit.client.0.smithi090.stdout: 15 16 227 211 0.0549411 0.152344 0.312299 1.10829 2021-05-12T12:22:22.509 INFO:tasks.workunit.client.0.smithi090.stdout: 16 16 299 283 0.0690833 0.28125 0.207404 0.895632 2021-05-12T12:22:23.510 INFO:tasks.workunit.client.0.smithi090.stdout: 17 16 362 346 0.0794938 0.246094 0.173022 0.780289 2021-05-12T12:22:24.510 INFO:tasks.workunit.client.0.smithi090.stdout: 18 16 432 416 0.0902664 0.273438 0.181435 0.686462 2021-05-12T12:22:25.510 INFO:tasks.workunit.client.0.smithi090.stdout: 19 16 497 481 0.0988773 0.253906 0.312053 0.629196 2021-05-12T12:22:26.510 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:22:26.508305+0000 min lat: 0.0583857 max lat: 13.4801 avg lat: 0.571492 2021-05-12T12:22:26.511 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:22:26.511 INFO:tasks.workunit.client.0.smithi090.stdout: 20 16 570 554 0.108189 0.285156 0.124058 0.571492 2021-05-12T12:22:27.510 INFO:tasks.workunit.client.0.smithi090.stdout: 21 16 629 613 0.114011 0.230469 0.190832 0.54285 2021-05-12T12:22:28.510 INFO:tasks.workunit.client.0.smithi090.stdout: 22 16 695 679 0.120546 0.257812 0.167709 0.514956 2021-05-12T12:22:29.511 INFO:tasks.workunit.client.0.smithi090.stdout: 23 16 765 749 0.127192 0.273438 0.187297 0.489525 2021-05-12T12:22:30.511 INFO:tasks.workunit.client.0.smithi090.stdout: 24 16 837 821 0.133609 0.28125 0.247088 0.464831 2021-05-12T12:22:31.511 INFO:tasks.workunit.client.0.smithi090.stdout: 25 16 902 886 0.13842 0.253906 0.335306 0.449358 2021-05-12T12:22:32.511 INFO:tasks.workunit.client.0.smithi090.stdout: 26 16 969 953 0.143161 0.261719 0.350606 0.434165 2021-05-12T12:22:33.511 INFO:tasks.workunit.client.0.smithi090.stdout: 27 16 1025 1009 0.145959 0.21875 0.148602 0.425827 2021-05-12T12:22:34.511 INFO:tasks.workunit.client.0.smithi090.stdout: 28 16 1087 1071 0.149394 0.242188 0.384997 0.415106 2021-05-12T12:22:35.511 INFO:tasks.workunit.client.0.smithi090.stdout: 29 16 1162 1146 0.154344 0.292969 0.265344 0.403879 2021-05-12T12:22:36.512 INFO:tasks.workunit.client.0.smithi090.stdout: 30 16 1233 1217 0.158443 0.277344 0.135812 0.393002 2021-05-12T12:22:37.512 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 30.0374 2021-05-12T12:22:37.512 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 1233 2021-05-12T12:22:37.512 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:22:37.513 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:22:37.513 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.160347 2021-05-12T12:22:37.513 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.121463 2021-05-12T12:22:37.513 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.292969 2021-05-12T12:22:37.514 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:22:37.514 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 41 2021-05-12T12:22:37.514 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 31.0946 2021-05-12T12:22:37.514 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 75 2021-05-12T12:22:37.514 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 0 2021-05-12T12:22:37.515 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.389677 2021-05-12T12:22:37.515 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 1.24548 2021-05-12T12:22:37.515 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 13.4801 2021-05-12T12:22:37.515 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.0583857 2021-05-12T12:22:37.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2021-05-12T12:22:37.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:22:37.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:22:37.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:22:37.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:22:37.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:22:37.568 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:22:37.568 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:22:37.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:22:37.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:22:37.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:22:37.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:22:37.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:22:37.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:22:37.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:22:37.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:22:38.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:22:38.019 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:22:38.019 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:22:38.020 INFO:tasks.workunit.client.0.smithi090.stderr:3' 2021-05-12T12:22:38.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:22:38.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:38.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:22:38.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=360777252876 2021-05-12T12:22:38.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 360777252876 2021-05-12T12:22:38.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-360777252876' 2021-05-12T12:22:38.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:38.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:22:38.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=373662154763 2021-05-12T12:22:38.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 373662154763 2021-05-12T12:22:38.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-360777252876 1-373662154763' 2021-05-12T12:22:38.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:38.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:22:38.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=390842023946 2021-05-12T12:22:38.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 390842023946 2021-05-12T12:22:38.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-360777252876 1-373662154763 2-390842023946' 2021-05-12T12:22:38.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:22:38.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:22:38.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=408021893129 2021-05-12T12:22:38.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 408021893129 2021-05-12T12:22:38.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-360777252876 1-373662154763 2-390842023946 3-408021893129' 2021-05-12T12:22:38.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:38.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-360777252876 2021-05-12T12:22:38.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:38.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:22:38.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-360777252876 2021-05-12T12:22:38.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:38.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=360777252876 2021-05-12T12:22:38.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 360777252876' 2021-05-12T12:22:38.530 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 360777252876 2021-05-12T12:22:38.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:22:38.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 360777252876 -lt 360777252876 2021-05-12T12:22:38.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:38.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-373662154763 2021-05-12T12:22:38.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:38.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:22:38.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-373662154763 2021-05-12T12:22:38.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:38.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=373662154763 2021-05-12T12:22:38.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 373662154763' 2021-05-12T12:22:38.929 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 373662154763 2021-05-12T12:22:38.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:22:39.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 373662154763 -lt 373662154763 2021-05-12T12:22:39.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:39.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-390842023946 2021-05-12T12:22:39.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:39.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:22:39.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-390842023946 2021-05-12T12:22:39.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:39.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=390842023946 2021-05-12T12:22:39.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 390842023946' 2021-05-12T12:22:39.305 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 390842023946 2021-05-12T12:22:39.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:22:39.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 390842023946 -lt 390842023946 2021-05-12T12:22:39.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:22:39.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-408021893129 2021-05-12T12:22:39.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:22:39.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:22:39.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-408021893129 2021-05-12T12:22:39.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:22:39.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=408021893129 2021-05-12T12:22:39.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 408021893129' 2021-05-12T12:22:39.664 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 408021893129 2021-05-12T12:22:39.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:22:40.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 408021893129 -lt 408021893129 2021-05-12T12:22:40.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:22:40.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:22:40.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:22:40.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:22:40.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:22:40.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:22:40.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:22:40.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:22:40.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:22:40.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:22:40.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:22:40.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:22:40.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:22:40.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:22:40.840 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:22:41.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:22:41.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:22:41.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:22:41.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2021-05-12T12:22:41.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:22:41.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:41.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:22:41.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:41.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:41.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:41.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:41.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:41.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:41.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:41.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:41.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:41.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:41.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:41.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:41.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:41.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:41.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:41.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:41.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:22:41.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:41.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:22:41.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94350 2021-05-12T12:22:41.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:41.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:41.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:22:41.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bluefs-volume-ops 0 2021-05-12T12:22:41.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T12:22:41.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:22:41.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T12:22:41.798 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:41.799 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:41.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:41.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:41.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:41.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:41.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:41.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:41.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:41.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:41.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:41.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:41.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:22:41.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T12:22:41.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:41.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94350 2021-05-12T12:22:41.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94350 2021-05-12T12:22:41.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:41.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:41.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:22:41.808 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:22:41.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:22:41.879 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2021-05-12T12:22:41.880 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2021-05-12T12:22:41.939 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:22:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:22:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:22:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:22:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:22:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:22:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:22:41.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:22:41.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:22:41.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-config 2021-05-12T12:22:41.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2021-05-12T12:22:41.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2021-05-12T12:22:41.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2021-05-12T12:22:41.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2021-05-12T12:22:41.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2021-05-12T12:22:41.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2021-05-12T12:22:41.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none ' 2021-05-12T12:22:41.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:41.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2021-05-12T12:22:41.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:22:41.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2021-05-12T12:22:41.955 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_config_track 2021-05-12T12:22:41.955 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_default_adjustment' 2021-05-12T12:22:41.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:22:41.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:22:41.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:22:41.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:22:41.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:41.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:41.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:41.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:41.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:41.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:41.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:41.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:41.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:41.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:41.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:41.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:41.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:41.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:41.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:41.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:41.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:41.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:41.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:41.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:41.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:41.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:41.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:41.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:22:41.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:41.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:41.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:22:41.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:41.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:41.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:41.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141126 2021-05-12T12:22:41.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:41.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:22:41.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:22:41.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:41.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:22:41.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2021-05-12T12:22:41.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2021-05-12T12:22:41.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2021-05-12T12:22:41.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T12:22:41.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:22:41.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:22:41.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:22:41.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T12:22:41.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T12:22:42.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:22:42.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:42.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:42.336 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:42.336 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:42.337 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:42.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:42.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:22:42.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:22:42.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:22:42.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:42.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:42.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:22:42.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:22:42.589 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:42.590 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:42.590 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:42.590 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:42.590 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:42.591 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:42.591 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-mon.a.asok 2021-05-12T12:22:42.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:42.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141126/ceph-mon.a.asok config get fsid 2021-05-12T12:22:42.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:22:42.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:42.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:42.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:22:42.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:22:42.684 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:42.684 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:42.686 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-mon.a.asok 2021-05-12T12:22:42.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:42.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141126/ceph-mon.a.asok config get mon_host 2021-05-12T12:22:42.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2021-05-12T12:22:42.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T12:22:42.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:22:42.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:22:42.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:22:42.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T12:22:42.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:22:43.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:22:43.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:43.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:43.641 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:43.641 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:43.641 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:43.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:43.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:43.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:43.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2021-05-12T12:22:43.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2021-05-12T12:22:43.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T12:22:43.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T12:22:43.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:22:43.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:22:43.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:22:43.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T12:22:43.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:43.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:43.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:43.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:43.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T12:22:43.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T12:22:43.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:43.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:22:43.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T12:22:43.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:22:43.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:43.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:43.669 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:43.670 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:43.670 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:43.670 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:43.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:43.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:43.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:22:43.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:22:43.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T12:22:43.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T12:22:43.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:43.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:43.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:43.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:22:43.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2021-05-12T12:22:43.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T12:22:43.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:22:43.673 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 0141954e-c7b8-4877-8efd-c70fc09ab916 2021-05-12T12:22:43.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0141954e-c7b8-4877-8efd-c70fc09ab916 2021-05-12T12:22:43.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 0141954e-c7b8-4877-8efd-c70fc09ab916' 2021-05-12T12:22:43.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:22:43.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCTyJtgUjg2KBAAOQCKitg9mkc5peOr8dQg6g== 2021-05-12T12:22:43.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCTyJtgUjg2KBAAOQCKitg9mkc5peOr8dQg6g=="}' 2021-05-12T12:22:43.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0141954e-c7b8-4877-8efd-c70fc09ab916 -i td/osd-config/0/new.json 2021-05-12T12:22:44.019 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:22:44.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T12:22:44.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQCTyJtgUjg2KBAAOQCKitg9mkc5peOr8dQg6g== --osd-uuid 0141954e-c7b8-4877-8efd-c70fc09ab916 2021-05-12T12:22:44.052 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.048+0000 7f7f8da1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:44.054 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.050+0000 7f7f8da1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:44.056 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.052+0000 7f7f8da1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:44.057 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.053+0000 7f7f8da1b0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:44.057 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.053+0000 7f7f8da1b0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:44.057 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.053+0000 7f7f8da1b0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:44.094 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:44.090+0000 7f7f8da1b0c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T12:22:48.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T12:22:48.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:22:48.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:22:48.770 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:22:48.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:22:49.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:22:49.239 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:22:49.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T12:22:49.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:22:49.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:22:49.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:22:49.258 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:49.254+0000 7fc9e49b10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:49.260 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:49.256+0000 7fc9e49b10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:49.262 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:49.258+0000 7fc9e49b10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:49.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:22:49.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:22:49.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:22:49.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:22:49.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:22:49.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:49.599 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:22:49.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:22:49.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:49.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:49.773+0000 7fc9e49b10c0 -1 Falling back to public interface 2021-05-12T12:22:49.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:50.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:50.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:50.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:22:50.976 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:22:50.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:50.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:51.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:22:51.995 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:51.991+0000 7fc9e49b10c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:22:52.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:22:52.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:22:52.330 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:22:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:22:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:22:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:22:52.693 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/142246,v1:127.0.0.1:6803/142246] [v2:127.0.0.1:6804/142246,v1:127.0.0.1:6805/142246] exists,up 0141954e-c7b8-4877-8efd-c70fc09ab916 2021-05-12T12:22:52.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:22:52.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:22:52.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:22:52.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2021-05-12T12:22:52.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:22:52.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:22:52.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:52.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:52.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-osd.0.asok 2021-05-12T12:22:52.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2021-05-12T12:22:52.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.141126/ceph-osd.0.asok log flush 2021-05-12T12:22:52.788 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:22:52.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:22:52.798 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:22:51.993+0000 7fc9dc13a700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T12:22:52.799 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:22:52.276+0000 7fc9e49b10c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T12:22:52.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:22:52.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.906 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.921 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:52.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:22:52.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:22:52.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:22:52.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:22:52.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:22:52.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:22:52.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:22:52.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:22:52.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:22:52.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:22:52.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:22:52.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:22:52.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:22:52.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:22:52.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:22:52.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:22:52.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:22:52.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:22:52.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:22:52.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:22:52.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:22:52.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:22:52.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:22:52.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:22:52.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:22:52.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:22:52.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:22:52.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:52.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:22:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:22:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:22:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:22:52.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:22:52.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:52.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:52.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141126 2021-05-12T12:22:52.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:22:52.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:22:52.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:22:52.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:22:52.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2021-05-12T12:22:52.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2021-05-12T12:22:52.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2021-05-12T12:22:52.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T12:22:52.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:22:52.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:22:52.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:22:52.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T12:22:52.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T12:22:53.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:22:53.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:53.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:53.283 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:53.283 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:53.283 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:53.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:53.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:22:53.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:22:53.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:22:53.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:53.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:53.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:22:53.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:22:53.694 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:53.695 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:53.695 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:53.695 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:53.695 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:53.696 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:53.696 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-mon.a.asok 2021-05-12T12:22:53.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:53.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141126/ceph-mon.a.asok config get fsid 2021-05-12T12:22:53.792 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:22:53.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:22:53.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:22:53.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:22:53.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:22:53.793 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:22:53.794 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:22:53.794 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:22:53.794 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:22:53.795 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:53.795 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:53.795 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-mon.a.asok 2021-05-12T12:22:53.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:22:53.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141126/ceph-mon.a.asok config get mon_host 2021-05-12T12:22:53.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2021-05-12T12:22:53.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T12:22:53.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:22:53.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:22:53.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:22:53.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T12:22:53.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:22:54.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:22:54.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:54.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:54.378 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:54.378 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:54.378 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:54.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:54.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:54.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:22:54.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2021-05-12T12:22:54.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T12:22:54.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:22:54.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:22:54.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:22:54.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T12:22:54.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T12:22:54.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:22:54.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:22:54.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:22:54.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T12:22:54.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T12:22:54.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:22:54.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:22:54.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T12:22:54.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:22:54.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:22:54.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:22:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:22:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:22:54.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:22:54.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' 2021-05-12T12:22:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:22:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:22:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:22:54.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T12:22:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T12:22:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:22:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:22:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:22:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:22:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T12:22:54.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:22:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=b889b357-275f-432f-a5f3-71daddde9f87 2021-05-12T12:22:54.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 b889b357-275f-432f-a5f3-71daddde9f87' 2021-05-12T12:22:54.410 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 b889b357-275f-432f-a5f3-71daddde9f87 2021-05-12T12:22:54.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:22:54.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCeyJtg7AGaGBAAlTlc9DKK49QhJXeBng1VBQ== 2021-05-12T12:22:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCeyJtg7AGaGBAAlTlc9DKK49QhJXeBng1VBQ=="}' 2021-05-12T12:22:54.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new b889b357-275f-432f-a5f3-71daddde9f87 -i td/osd-config/0/new.json 2021-05-12T12:22:54.724 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:22:54.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T12:22:54.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCeyJtg7AGaGBAAlTlc9DKK49QhJXeBng1VBQ== --osd-uuid b889b357-275f-432f-a5f3-71daddde9f87 2021-05-12T12:22:54.756 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.753+0000 7fc3731de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.758 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.754+0000 7fc3731de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.759 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.755+0000 7fc3731de0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:54.760 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.756+0000 7fc3731de0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:54.760 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.756+0000 7fc3731de0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:54.760 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.756+0000 7fc3731de0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T12:22:54.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:54.795+0000 7fc3731de0c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T12:22:59.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T12:22:59.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:22:59.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:22:59.379 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:22:59.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:22:59.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:22:59.853 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:22:59.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=fafff420-ccbb-49e2-a055-740c04abbd22 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141126/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:22:59.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:22:59.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:22:59.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:22:59.871 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:59.869+0000 7f08de86c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:59.873 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:59.870+0000 7f08de86c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:22:59.874 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:22:59.871+0000 7f08de86c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:00.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:23:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:23:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:00.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:00.208 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:00.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:00.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:00.399 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:00.396+0000 7f08de86c0c0 -1 Falling back to public interface 2021-05-12T12:23:00.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:01.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:01.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:01.567 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:01.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:01.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:01.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:01.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:02.586 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:02.583+0000 7f08de86c0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:23:02.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:02.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:02.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:23:02.924 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:23:02.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:02.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:03.289 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/143937,v1:127.0.0.1:6803/143937] [v2:127.0.0.1:6804/143937,v1:127.0.0.1:6805/143937] exists,up b889b357-275f-432f-a5f3-71daddde9f87 2021-05-12T12:23:03.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:03.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:03.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:03.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2021-05-12T12:23:03.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2021-05-12T12:23:03.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2021-05-12T12:23:03.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2021-05-12T12:23:03.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2021-05-12T12:23:03.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2021-05-12T12:23:03.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:23:03.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2021-05-12T12:23:03.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2021-05-12T12:23:03.427 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:23:03.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2021-05-12T12:23:03.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:23:03.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:23:03.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:03.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-osd.0.asok 2021-05-12T12:23:03.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2021-05-12T12:23:03.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.141126/ceph-osd.0.asok log flush 2021-05-12T12:23:03.531 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:23:03.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T12:23:03.540 INFO:tasks.workunit.client.0.smithi090.stdout:2021-05-12T12:23:03.413+0000 7f08d7ff9700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2021-05-12T12:23:03.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2021-05-12T12:23:03.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2021-05-12T12:23:03.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:23:03.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:23:03.543 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:03.543 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.543 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141126/ceph-osd.0.asok 2021-05-12T12:23:03.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2021-05-12T12:23:03.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.141126/ceph-osd.0.asok log reopen 2021-05-12T12:23:03.634 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:23:03.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:23:03.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:23:03.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:03.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:23:03.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:03.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:03.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:03.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:03.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:03.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:03.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:03.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:03.750 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:03.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:03.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:03.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:03.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:03.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:03.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:03.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:03.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:23:03.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:03.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:23:03.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:03.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:03.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T12:23:03.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T12:23:03.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T12:23:03.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T12:23:03.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:23:03.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:03.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:23:03.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:03.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:03.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:03.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:03.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:03.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:03.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:03.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:03.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:03.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:03.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:03.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:03.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:03.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:03.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:03.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:03.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:23:03.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:03.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:23:03.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:03.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:03.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T12:23:03.770 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:23:03.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141126 2021-05-12T12:23:03.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:23:03.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:23:03.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:23:03.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T12:23:03.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2021-05-12T12:23:03.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2021-05-12T12:23:03.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2021-05-12T12:23:03.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2021-05-12T12:23:03.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2021-05-12T12:23:03.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2021-05-12T12:23:03.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2021-05-12T12:23:03.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2021-05-12T12:23:03.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2021-05-12T12:23:03.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2021-05-12T12:23:03.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2021-05-12T12:23:03.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2021-05-12T12:23:03.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2021-05-12T12:23:03.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2021-05-12T12:23:03.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2021-05-12T12:23:03.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2021-05-12T12:23:03.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2021-05-12T12:23:03.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2021-05-12T12:23:03.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2021-05-12T12:23:03.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T12:23:03.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:23:03.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:03.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:23:03.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:03.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:03.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:03.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:03.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:03.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:03.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:03.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:03.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:03.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:03.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:03.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:03.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:03.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:03.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:03.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:03.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:23:03.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:03.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.857 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:23:03.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:03.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:03.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:23:03.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-config 0 2021-05-12T12:23:03.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T12:23:03.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:23:03.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T12:23:03.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:03.861 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:03.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:03.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:03.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:03.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:03.862 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:03.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:03.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:03.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:03.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:03.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:03.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:03.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:03.865 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:03.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:03.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:23:03.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T12:23:03.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:03.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:03.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141126 2021-05-12T12:23:03.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141126 2021-05-12T12:23:03.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:03.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:03.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:23:03.869 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:23:03.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:23:03.931 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2021-05-12T12:23:04.008 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2021-05-12T12:23:04.027 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:23:04.035 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:23:04.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:04.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:04.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:23:04.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:23:04.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-copy-from 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2021-05-12T12:23:04.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2021-05-12T12:23:04.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2021-05-12T12:23:04.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2021-05-12T12:23:04.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2021-05-12T12:23:04.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2021-05-12T12:23:04.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none ' 2021-05-12T12:23:04.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2021-05-12T12:23:04.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2021-05-12T12:23:04.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:23:04.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2021-05-12T12:23:04.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2021-05-12T12:23:04.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2021-05-12T12:23:04.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2021-05-12T12:23:04.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2021-05-12T12:23:04.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:23:04.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:04.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:23:04.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:04.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:04.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:04.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:04.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:04.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:04.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:04.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:04.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:04.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:04.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:04.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:04.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:04.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:04.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:04.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:23:04.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:04.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:04.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:04.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144634 2021-05-12T12:23:04.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:04.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:04.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2021-05-12T12:23:04.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:23:04.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:04.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:04.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.144634 2021-05-12T12:23:04.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:23:04.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:23:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:23:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:23:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2021-05-12T12:23:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:23:04.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:23:04.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2021-05-12T12:23:04.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2021-05-12T12:23:04.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:23:04.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:04.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:04.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:04.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:04.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:04.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:04.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:23:04.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:23:04.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:23:04.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:04.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:04.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:23:04.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:23:04.659 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:04.659 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:04.659 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:04.659 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:04.660 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:04.660 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:04.660 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.144634/ceph-mon.a.asok 2021-05-12T12:23:04.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:04.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.144634/ceph-mon.a.asok config get fsid 2021-05-12T12:23:04.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:23:04.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:04.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:04.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:23:04.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:23:04.754 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:04.754 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:04.754 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:04.755 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:04.755 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:04.755 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:04.755 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.144634/ceph-mon.a.asok 2021-05-12T12:23:04.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:04.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.144634/ceph-mon.a.asok config get mon_host 2021-05-12T12:23:04.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2021-05-12T12:23:04.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2021-05-12T12:23:04.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:23:04.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:23:04.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:23:04.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2021-05-12T12:23:04.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:23:05.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:23:05.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:05.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:05.711 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:05.711 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:05.711 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:05.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:05.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:05.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:05.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2021-05-12T12:23:05.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T12:23:05.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:23:05.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:23:05.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:23:05.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2021-05-12T12:23:05.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T12:23:05.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:05.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:05.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:05.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2021-05-12T12:23:05.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2021-05-12T12:23:05.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:23:05.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:23:05.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T12:23:05.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:23:05.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:05.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:05.740 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:05.740 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:05.740 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:05.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:05.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:05.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:05.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:05.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:05.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T12:23:05.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T12:23:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:23:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:23:05.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/0 2021-05-12T12:23:05.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:23:05.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=b7c3cc13-04ae-4543-9b7f-c3ef0be23977 2021-05-12T12:23:05.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 b7c3cc13-04ae-4543-9b7f-c3ef0be23977' 2021-05-12T12:23:05.744 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 b7c3cc13-04ae-4543-9b7f-c3ef0be23977 2021-05-12T12:23:05.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:23:05.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCpyJtgLdtsLBAAut7Bj4ADt5n3rlJEAggMFA== 2021-05-12T12:23:05.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCpyJtgLdtsLBAAut7Bj4ADt5n3rlJEAggMFA=="}' 2021-05-12T12:23:05.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new b7c3cc13-04ae-4543-9b7f-c3ef0be23977 -i td/osd-copy-from/0/new.json 2021-05-12T12:23:06.081 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:06.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/0/new.json 2021-05-12T12:23:06.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCpyJtgLdtsLBAAut7Bj4ADt5n3rlJEAggMFA== --osd-uuid b7c3cc13-04ae-4543-9b7f-c3ef0be23977 2021-05-12T12:23:06.114 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.111+0000 7ff8c03830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:06.116 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.113+0000 7ff8c03830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:06.118 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.115+0000 7ff8c03830c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:06.118 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.115+0000 7ff8c03830c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:23:06.119 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.115+0000 7ff8c03830c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:23:06.119 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.115+0000 7ff8c03830c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T12:23:06.156 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:06.153+0000 7ff8c03830c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2021-05-12T12:23:10.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/0/keyring 2021-05-12T12:23:10.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:23:10.495 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:23:10.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:23:10.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:10.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:23:10.977 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:23:10.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:10.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:23:10.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:23:10.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:23:10.996 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:10.993+0000 7f374caae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:10.997 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:10.995+0000 7f374caae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:10.998 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:10.996+0000 7f374caae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:11.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:23:11.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:11.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:23:11.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:11.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:11.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:11.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:11.335 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:11.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:11.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:11.514 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:11.512+0000 7f374caae0c0 -1 Falling back to public interface 2021-05-12T12:23:11.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:12.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:12.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:12.687 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:12.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:12.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:12.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:13.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:13.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:13.698+0000 7f374caae0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:23:14.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:14.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:14.045 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:23:14.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:23:14.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:14.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:14.406 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/145754,v1:127.0.0.1:6803/145754] [v2:127.0.0.1:6804/145754,v1:127.0.0.1:6805/145754] exists,up b7c3cc13-04ae-4543-9b7f-c3ef0be23977 2021-05-12T12:23:14.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:14.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:14.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:14.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2021-05-12T12:23:14.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T12:23:14.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:23:14.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:23:14.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:23:14.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2021-05-12T12:23:14.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T12:23:14.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:14.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:14.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:14.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2021-05-12T12:23:14.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2021-05-12T12:23:14.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:23:14.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:23:14.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T12:23:14.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:23:14.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:14.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:14.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:14.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:14.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:14.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:14.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' 2021-05-12T12:23:14.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:14.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:14.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:14.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T12:23:14.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T12:23:14.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:14.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:14.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:14.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:23:14.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:23:14.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/1 2021-05-12T12:23:14.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:23:14.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f150fe59-a030-4c82-8775-a80c41c92f08 2021-05-12T12:23:14.418 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 f150fe59-a030-4c82-8775-a80c41c92f08 2021-05-12T12:23:14.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f150fe59-a030-4c82-8775-a80c41c92f08' 2021-05-12T12:23:14.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:23:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCyyJtg6t/3GBAAV0RtrRtxmFyz2ZruYVUXGg== 2021-05-12T12:23:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCyyJtg6t/3GBAAV0RtrRtxmFyz2ZruYVUXGg=="}' 2021-05-12T12:23:14.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f150fe59-a030-4c82-8775-a80c41c92f08 -i td/osd-copy-from/1/new.json 2021-05-12T12:23:14.945 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:14.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/1/new.json 2021-05-12T12:23:14.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCyyJtg6t/3GBAAV0RtrRtxmFyz2ZruYVUXGg== --osd-uuid f150fe59-a030-4c82-8775-a80c41c92f08 2021-05-12T12:23:14.979 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:14.976+0000 7f64f39c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:15.004 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.001+0000 7f64f39c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:15.004 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.002+0000 7f64f39c90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:15.005 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.002+0000 7f64f39c90c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:15.005 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.002+0000 7f64f39c90c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:15.006 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.002+0000 7f64f39c90c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T12:23:15.103 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:15.101+0000 7f64f39c90c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2021-05-12T12:23:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/1/keyring 2021-05-12T12:23:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:23:19.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:23:19.683 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:23:19.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:20.165 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:23:20.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:23:20.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=29216fb6-3c47-45d3-b29c-f9b0f256b61e --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144634/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:20.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:23:20.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:23:20.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:23:20.184 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:20.181+0000 7f06b6ac50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.186 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:20.183+0000 7f06b6ac50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.187 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:20.184+0000 7f06b6ac50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:20.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:23:20.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:20.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:23:20.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:20.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:20.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:20.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:20.525 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:20.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:20.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:20.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:20.699+0000 7f06b6ac50c0 -1 Falling back to public interface 2021-05-12T12:23:20.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:21.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:21.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:21.902 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:21.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:21.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:21.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:22.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:22.941 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:22.938+0000 7f06b6ac50c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:23:23.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:23.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:23.279 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:23:23.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:23:23.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:23.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:23.661 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147139,v1:127.0.0.1:6811/147139] [v2:127.0.0.1:6812/147139,v1:127.0.0.1:6813/147139] exists,up f150fe59-a030-4c82-8775-a80c41c92f08 2021-05-12T12:23:23.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:23.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:23.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:23.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2021-05-12T12:23:23.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:23:24.311 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' does not exist 2021-05-12T12:23:24.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:23:24.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:23:24.869 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' already exists 2021-05-12T12:23:24.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:23:25.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:23:27.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2021-05-12T12:23:27.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2021-05-12T12:23:27.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2021-05-12T12:23:27.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2021-05-12T12:23:27.374 INFO:tasks.workunit.client.0.smithi090.stdout:rbd/foo2 mtime 2021-05-12T12:23:27.000000+0000, size 825480 2021-05-12T12:23:27.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2021-05-12T12:23:27.533 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0: {} 2021-05-12T12:23:27.563 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1: {} 2021-05-12T12:23:27.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T12:23:27.596 INFO:tasks.workunit.client.0.smithi090.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2021-05-12T12:23:27.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T12:23:27.619 INFO:tasks.workunit.client.0.smithi090.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2021-05-12T12:23:27.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2021-05-12T12:23:27.741 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0: {} 2021-05-12T12:23:27.770 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1: {} 2021-05-12T12:23:27.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T12:23:27.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T12:23:27.939 INFO:tasks.workunit.client.0.smithi090.stdout:rbd/foo3 mtime 2021-05-12T12:23:27.000000+0000, size 825480 2021-05-12T12:23:27.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2021-05-12T12:23:27.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:23:27.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:27.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:23:27.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:27.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:27.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:27.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:27.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:28.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:28.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:28.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:28.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:28.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:28.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:28.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:28.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:28.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:28.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:28.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:28.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:23:28.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:28.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:28.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144634 2021-05-12T12:23:28.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:28.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:28.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:23:28.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-copy-from 0 2021-05-12T12:23:28.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T12:23:28.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:23:28.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T12:23:28.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:28.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:28.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:28.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:28.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:28.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:28.064 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:28.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:28.064 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:28.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:28.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:28.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:28.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:28.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:28.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:28.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:23:28.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T12:23:28.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:28.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144634 2021-05-12T12:23:28.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144634 2021-05-12T12:23:28.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:28.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:28.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:23:28.071 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:23:28.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:23:28.132 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2021-05-12T12:23:28.133 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2021-05-12T12:23:28.192 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:23:28.201 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:23:28.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:28.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:23:28.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:23:28.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:23:28.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:23:28.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:23:28.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:23:28.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:23:28.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-dup 2021-05-12T12:23:28.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2021-05-12T12:23:28.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2021-05-12T12:23:28.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T12:23:28.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T12:23:28.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2021-05-12T12:23:28.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2021-05-12T12:23:28.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none ' 2021-05-12T12:23:28.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T12:23:28.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2021-05-12T12:23:28.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:28.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2021-05-12T12:23:28.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:23:28.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2021-05-12T12:23:28.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2021-05-12T12:23:28.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2021-05-12T12:23:28.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2021-05-12T12:23:28.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2021-05-12T12:23:28.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:23:28.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:23:28.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:23:28.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:23:28.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:23:28.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:23:28.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:23:28.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:23:28.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:23:28.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:23:28.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:23:28.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:23:28.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:23:28.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:23:28.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:23:28.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:23:28.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:23:28.213 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:23:28.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:23:28.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:23:28.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:23:28.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:23:28.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:28.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148024 2021-05-12T12:23:28.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:23:28.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:23:28.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2021-05-12T12:23:28.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:23:28.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:28.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.148024 2021-05-12T12:23:28.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:23:28.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:23:28.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:23:28.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:23:28.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2021-05-12T12:23:28.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2021-05-12T12:23:28.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2021-05-12T12:23:28.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2021-05-12T12:23:28.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2021-05-12T12:23:28.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T12:23:28.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2021-05-12T12:23:28.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2021-05-12T12:23:28.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:23:28.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:23:28.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:23:28.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2021-05-12T12:23:28.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2021-05-12T12:23:28.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:23:28.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:28.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:28.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:28.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:23:28.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:23:28.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:23:28.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:28.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:28.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:23:28.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:23:28.803 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:28.803 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:28.804 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:28.804 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:28.804 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.804 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:28.805 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.148024/ceph-mon.a.asok 2021-05-12T12:23:28.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:28.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.148024/ceph-mon.a.asok config get fsid 2021-05-12T12:23:28.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:23:28.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:23:28.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:23:28.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:23:28.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:23:28.897 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:23:28.897 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:23:28.898 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:23:28.898 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:23:28.898 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:28.898 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:28.899 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.148024/ceph-mon.a.asok 2021-05-12T12:23:28.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:23:28.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.148024/ceph-mon.a.asok config get mon_host 2021-05-12T12:23:28.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2021-05-12T12:23:28.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2021-05-12T12:23:28.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:23:28.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:23:28.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:23:28.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2021-05-12T12:23:28.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:23:29.854 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:23:29.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:29.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:29.855 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:29.856 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:29.856 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:29.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:29.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:29.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:23:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2021-05-12T12:23:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:29.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=0 2021-05-12T12:23:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/0 2021-05-12T12:23:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:29.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:29.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:29.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T12:23:29.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T12:23:29.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:29.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:29.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:29.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:29.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:29.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:29.885 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:29.885 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:29.885 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:29.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:29.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:29.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:29.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:29.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:29.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:29.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/0 2021-05-12T12:23:29.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:29.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:23:29.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd a9a088d3-b068-4596-b50a-7a4da8bcd5ad' 2021-05-12T12:23:29.890 INFO:tasks.workunit.client.0.smithi090.stdout:add osd a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:23:29.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:29.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQDByJtg4fYeNRAAYMY0QqsY033ZqkjGhcR3DA== 2021-05-12T12:23:29.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQDByJtg4fYeNRAAYMY0QqsY033ZqkjGhcR3DA=="}' 2021-05-12T12:23:29.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new a9a088d3-b068-4596-b50a-7a4da8bcd5ad -i td/osd-dup/0/new.json 2021-05-12T12:23:30.210 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:30.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/0/new.json 2021-05-12T12:23:30.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 0 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDByJtg4fYeNRAAYMY0QqsY033ZqkjGhcR3DA== --osd-uuid a9a088d3-b068-4596-b50a-7a4da8bcd5ad --osd-objectstore=filestore 2021-05-12T12:23:30.244 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:30.241+0000 7f91abad60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:30.246 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:30.243+0000 7f91abad60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:30.248 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:30.244+0000 7f91abad60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:30.604 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:30.601+0000 7f91abad60c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:31.251 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.248+0000 7f91abad60c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:31.256 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.253+0000 7f91abad60c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:31.256 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.253+0000 7f91abad60c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:31.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2021-05-12T12:23:31.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:31.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd0 key to auth repository 2021-05-12T12:23:31.661 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:23:31.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:31.837 INFO:tasks.workunit.client.0.smithi090.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-05-12T12:23:31.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.0 2021-05-12T12:23:31.839 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:23:31.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 0 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:31.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:31.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:31.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:31.860 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.857+0000 7fe4c4f280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:31.862 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.859+0000 7fe4c4f280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:31.862 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.860+0000 7fe4c4f280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:31.864 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:31.861+0000 7fe4c4f280c0 -1 Falling back to public interface 2021-05-12T12:23:32.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 0 2021-05-12T12:23:32.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:32.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:23:32.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:32.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:32.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:32.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:32.230 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:32.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:32.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:32.382 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:32.380+0000 7fe4c4f280c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:32.389 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:32.386+0000 7fe4c4f280c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:32.389 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:32.386+0000 7fe4c4f280c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:32.397 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:32.394+0000 7fe4c4f280c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:23:32.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:33.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:33.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:33.449 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:33.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:23:33.661 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/148347,v1:127.0.0.1:6801/148347] [v2:127.0.0.1:6802/148347,v1:127.0.0.1:6803/148347] exists,up a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:23:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:33.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2021-05-12T12:23:33.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=148347 2021-05-12T12:23:33.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2021-05-12T12:23:33.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:33.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:33.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=1 2021-05-12T12:23:33.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:33.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/1 2021-05-12T12:23:33.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:33.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2021-05-12T12:23:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2021-05-12T12:23:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:33.670 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:33.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:33.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:33.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:33.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:33.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:33.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:33.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:33.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:33.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:33.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:33.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:33.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:33.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:33.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:33.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/1 2021-05-12T12:23:33.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:33.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=5fc58d60-3d58-4cac-b036-be68b5964c23 2021-05-12T12:23:33.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd 5fc58d60-3d58-4cac-b036-be68b5964c23' 2021-05-12T12:23:33.675 INFO:tasks.workunit.client.0.smithi090.stdout:add osd 5fc58d60-3d58-4cac-b036-be68b5964c23 2021-05-12T12:23:33.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:33.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQDFyJtgACtPKBAAkzpo0gU+EtXBEOISQqtS1A== 2021-05-12T12:23:33.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQDFyJtgACtPKBAAkzpo0gU+EtXBEOISQqtS1A=="}' 2021-05-12T12:23:33.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new 5fc58d60-3d58-4cac-b036-be68b5964c23 -i td/osd-dup/1/new.json 2021-05-12T12:23:34.013 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:34.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/1/new.json 2021-05-12T12:23:34.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 1 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDFyJtgACtPKBAAkzpo0gU+EtXBEOISQqtS1A== --osd-uuid 5fc58d60-3d58-4cac-b036-be68b5964c23 --osd-objectstore=filestore 2021-05-12T12:23:34.047 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:34.045+0000 7f35ed5060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:34.049 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:34.047+0000 7f35ed5060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:34.050 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:34.047+0000 7f35ed5060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:34.508 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:34.505+0000 7f35ed5060c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:35.293 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:35.290+0000 7f35ed5060c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:35.315 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:35.312+0000 7f35ed5060c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:35.315 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:35.312+0000 7f35ed5060c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:35.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2021-05-12T12:23:35.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:35.887 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:23:35.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd1 key to auth repository 2021-05-12T12:23:35.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:36.205 INFO:tasks.workunit.client.0.smithi090.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-05-12T12:23:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.1 2021-05-12T12:23:36.207 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:23:36.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 1 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:36.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:36.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:36.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:36.227 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.224+0000 7fc67c0160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:36.229 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.226+0000 7fc67c0160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:36.230 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.228+0000 7fc67c0160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:36.232 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.229+0000 7fc67c0160c0 -1 Falling back to public interface 2021-05-12T12:23:36.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 1 2021-05-12T12:23:36.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:36.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:23:36.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:36.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:36.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:36.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:36.727 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:36.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:36.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:36.830 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.827+0000 7fc67c0160c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.837+0000 7fc67c0160c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.837+0000 7fc67c0160c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:36.847 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:36.845+0000 7fc67c0160c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:23:37.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:38.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:38.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:38.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:38.130 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:38.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:38.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:23:38.485 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/148727,v1:127.0.0.1:6811/148727] [v2:127.0.0.1:6812/148727,v1:127.0.0.1:6813/148727] exists,up 5fc58d60-3d58-4cac-b036-be68b5964c23 2021-05-12T12:23:38.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:38.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:38.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:38.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2021-05-12T12:23:38.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T12:23:38.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T12:23:38.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=2 2021-05-12T12:23:38.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T12:23:38.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/2 2021-05-12T12:23:38.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:23:38.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:23:38.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:23:38.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:23:38.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2021-05-12T12:23:38.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2021-05-12T12:23:38.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T12:23:38.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T12:23:38.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:23:38.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T12:23:38.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:23:38.495 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:23:38.495 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:23:38.495 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:23:38.495 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:23:38.495 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:38.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:23:38.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T12:23:38.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T12:23:38.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T12:23:38.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:23:38.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:23:38.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:23:38.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:23:38.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:38.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T12:23:38.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T12:23:38.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/2 2021-05-12T12:23:38.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T12:23:38.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=1e315773-b6fe-4df4-86b8-284ed86f2910 2021-05-12T12:23:38.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd 1e315773-b6fe-4df4-86b8-284ed86f2910' 2021-05-12T12:23:38.499 INFO:tasks.workunit.client.0.smithi090.stdout:add osd 1e315773-b6fe-4df4-86b8-284ed86f2910 2021-05-12T12:23:38.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T12:23:38.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQDKyJtgV8bcHRAAhcaQ8IXg2LLOFVeuc5CdNQ== 2021-05-12T12:23:38.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQDKyJtgV8bcHRAAhcaQ8IXg2LLOFVeuc5CdNQ=="}' 2021-05-12T12:23:38.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new 1e315773-b6fe-4df4-86b8-284ed86f2910 -i td/osd-dup/2/new.json 2021-05-12T12:23:38.959 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:23:38.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/2/new.json 2021-05-12T12:23:38.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 2 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKyJtgV8bcHRAAhcaQ8IXg2LLOFVeuc5CdNQ== --osd-uuid 1e315773-b6fe-4df4-86b8-284ed86f2910 --osd-objectstore=filestore 2021-05-12T12:23:38.994 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:38.990+0000 7fe2c5d860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:38.995 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:38.992+0000 7fe2c5d860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:38.997 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:38.993+0000 7fe2c5d860c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:39.382 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:39.379+0000 7fe2c5d860c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:39.853 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:39.849+0000 7fe2c5d860c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:39.862 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:39.859+0000 7fe2c5d860c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:39.862 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:39.859+0000 7fe2c5d860c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T12:23:40.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2021-05-12T12:23:40.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T12:23:40.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd2 key to auth repository 2021-05-12T12:23:40.326 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:23:40.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:23:40.650 INFO:tasks.workunit.client.0.smithi090.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-05-12T12:23:40.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.2 2021-05-12T12:23:40.653 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:23:40.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 2 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:23:40.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T12:23:40.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T12:23:40.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T12:23:40.672 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:40.669+0000 7ff475eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:40.676 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:40.672+0000 7ff475eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:40.677 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:40.674+0000 7ff475eb50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:23:40.678 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:40.675+0000 7ff475eb50c0 -1 Falling back to public interface 2021-05-12T12:23:41.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 2 2021-05-12T12:23:41.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:23:41.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:23:41.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:23:41.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:23:41.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:41.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:23:41.013 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:23:41.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:41.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:23:41.127 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:41.123+0000 7ff475eb50c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T12:23:41.137 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:41.133+0000 7ff475eb50c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:41.137 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:41.133+0000 7ff475eb50c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T12:23:41.143 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:23:41.139+0000 7ff475eb50c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:23:41.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:23:42.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:23:42.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:23:42.375 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:23:42.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:23:42.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:23:42.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:23:42.815 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149019,v1:127.0.0.1:6819/149019] [v2:127.0.0.1:6820/149019,v1:127.0.0.1:6821/149019] exists,up 1e315773-b6fe-4df4-86b8-284ed86f2910 2021-05-12T12:23:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:23:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:23:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:23:42.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2021-05-12T12:23:47.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2021-05-12T12:23:47.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T12:23:48.317 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'foo' already exists 2021-05-12T12:23:48.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:23:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2021-05-12T12:23:49.358 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:23:49.358 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2021-05-12T12:23:49.359 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_149208 2021-05-12T12:23:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:23:49.359 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:23:50.357 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 16 0 0 0 - 0 2021-05-12T12:23:51.357 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 16 0 0 0 - 0 2021-05-12T12:23:52.357 INFO:tasks.workunit.client.0.smithi090.stdout: 3 16 23 7 0.00911344 0.00911458 0.175516 2.35048 2021-05-12T12:23:53.357 INFO:tasks.workunit.client.0.smithi090.stdout: 4 16 23 7 0.00683503 0 - 2.35048 2021-05-12T12:23:54.357 INFO:tasks.workunit.client.0.smithi090.stdout: 5 16 23 7 0.00546801 0 - 2.35048 2021-05-12T12:23:55.358 INFO:tasks.workunit.client.0.smithi090.stdout: 6 16 23 7 0.00455667 0 - 2.35048 2021-05-12T12:23:56.358 INFO:tasks.workunit.client.0.smithi090.stdout: 7 16 119 103 0.0574702 0.09375 0.120467 1.08301 2021-05-12T12:23:57.358 INFO:tasks.workunit.client.0.smithi090.stdout: 8 16 195 179 0.0873911 0.296875 0.463806 0.706894 2021-05-12T12:23:58.358 INFO:tasks.workunit.client.0.smithi090.stdout: 9 16 295 279 0.121079 0.390625 0.226738 0.514699 2021-05-12T12:23:59.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16 344 328 0.128109 0.191406 0.143087 0.483052 2021-05-12T12:24:00.358 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 10.0896 2021-05-12T12:24:00.359 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 344 2021-05-12T12:24:00.359 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:24:00.359 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:24:00.360 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.133182 2021-05-12T12:24:00.360 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.145237 2021-05-12T12:24:00.360 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.390625 2021-05-12T12:24:00.360 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0 2021-05-12T12:24:00.360 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 34 2021-05-12T12:24:00.361 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 37.2035 2021-05-12T12:24:00.361 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 100 2021-05-12T12:24:00.361 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 0 2021-05-12T12:24:00.361 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.466923 2021-05-12T12:24:00.362 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 1.15879 2021-05-12T12:24:00.362 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 6.32585 2021-05-12T12:24:00.362 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.0940396 2021-05-12T12:24:00.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 148347 2021-05-12T12:24:00.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2021-05-12T12:24:01.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 148347 2021-05-12T12:24:01.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (148347) - No such process 2021-05-12T12:24:01.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2021-05-12T12:24:01.837 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already down. 2021-05-12T12:24:01.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2021-05-12T12:24:01.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2021-05-12T12:24:01.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2021-05-12T12:24:01.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:24:01.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid a9a088d3-b068-4596-b50a-7a4da8bcd5ad' 2021-05-12T12:24:01.850 INFO:tasks.workunit.client.0.smithi090.stdout:osd fsid a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:24:01.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:24:01.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2021-05-12T12:24:01.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid a9a088d3-b068-4596-b50a-7a4da8bcd5ad --op mkfs --no-mon-config 2021-05-12T12:24:05.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2021-05-12T12:24:05.861 INFO:tasks.workunit.client.0.smithi090.stdout:dup from filestore: td/osd-dup/0.old 2021-05-12T12:24:05.900 INFO:tasks.workunit.client.0.smithi090.stdout: to bluestore: td/osd-dup/0 2021-05-12T12:24:06.867 INFO:tasks.workunit.client.0.smithi090.stdout:fsid a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:24:06.867 INFO:tasks.workunit.client.0.smithi090.stdout:17 collections 2021-05-12T12:24:06.868 INFO:tasks.workunit.client.0.smithi090.stdout:1/17 1.3_head 2021-05-12T12:24:07.077 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 24 objects, 94208 bytes, 29 keys 2021-05-12T12:24:07.078 INFO:tasks.workunit.client.0.smithi090.stdout:2/17 1.b_head 2021-05-12T12:24:07.261 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2021-05-12T12:24:07.262 INFO:tasks.workunit.client.0.smithi090.stdout:3/17 1.a_head 2021-05-12T12:24:07.372 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 23 objects, 90112 bytes, 28 keys 2021-05-12T12:24:07.373 INFO:tasks.workunit.client.0.smithi090.stdout:4/17 1.e_head 2021-05-12T12:24:07.531 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 20 objects, 77824 bytes, 25 keys 2021-05-12T12:24:07.532 INFO:tasks.workunit.client.0.smithi090.stdout:5/17 1.c_head 2021-05-12T12:24:07.715 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2021-05-12T12:24:07.715 INFO:tasks.workunit.client.0.smithi090.stdout:6/17 1.f_head 2021-05-12T12:24:07.860 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2021-05-12T12:24:07.860 INFO:tasks.workunit.client.0.smithi090.stdout:7/17 1.8_head 2021-05-12T12:24:08.019 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 23 objects, 90112 bytes, 28 keys 2021-05-12T12:24:08.019 INFO:tasks.workunit.client.0.smithi090.stdout:8/17 1.7_head 2021-05-12T12:24:08.179 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 23 objects, 90112 bytes, 28 keys 2021-05-12T12:24:08.179 INFO:tasks.workunit.client.0.smithi090.stdout:9/17 1.1_head 2021-05-12T12:24:08.330 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2021-05-12T12:24:08.331 INFO:tasks.workunit.client.0.smithi090.stdout:10/17 meta 2021-05-12T12:24:08.333 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 47 objects, 49250 bytes, 0 keys 2021-05-12T12:24:08.334 INFO:tasks.workunit.client.0.smithi090.stdout:11/17 1.d_head 2021-05-12T12:24:08.474 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 22 objects, 81944 bytes, 27 keys 2021-05-12T12:24:08.474 INFO:tasks.workunit.client.0.smithi090.stdout:12/17 1.5_head 2021-05-12T12:24:08.649 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 19 objects, 73728 bytes, 24 keys 2021-05-12T12:24:08.649 INFO:tasks.workunit.client.0.smithi090.stdout:13/17 1.6_head 2021-05-12T12:24:08.758 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 15 objects, 57344 bytes, 20 keys 2021-05-12T12:24:08.758 INFO:tasks.workunit.client.0.smithi090.stdout:14/17 1.4_head 2021-05-12T12:24:08.873 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 21 objects, 81920 bytes, 26 keys 2021-05-12T12:24:08.874 INFO:tasks.workunit.client.0.smithi090.stdout:15/17 1.9_head 2021-05-12T12:24:09.036 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 16 objects, 61440 bytes, 21 keys 2021-05-12T12:24:09.036 INFO:tasks.workunit.client.0.smithi090.stdout:16/17 1.2_head 2021-05-12T12:24:09.037 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-05-12T12:24:09.037 INFO:tasks.workunit.client.0.smithi090.stdout:17/17 1.0_head 2021-05-12T12:24:09.194 INFO:tasks.workunit.client.0.smithi090.stdout: 0 objects, 0 bytes, 0 keys 23 objects, 90112 bytes, 28 keys 2021-05-12T12:24:09.195 INFO:tasks.workunit.client.0.smithi090.stdout:keyring 2021-05-12T12:24:09.195 INFO:tasks.workunit.client.0.smithi090.stdout:duping osd metadata 2021-05-12T12:24:09.649 INFO:tasks.workunit.client.0.smithi090.stdout:done. 2021-05-12T12:24:10.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:24:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2021-05-12T12:24:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-dup 2021-05-12T12:24:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:24:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:24:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:24:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-dup/0 2021-05-12T12:24:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T12:24:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T12:24:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T12:24:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:10.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:24:10.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T12:24:10.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:24:10.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:10.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:10.432 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:10.432 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:10.432 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:24:10.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:24:10.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' 2021-05-12T12:24:10.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:10.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T12:24:10.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T12:24:10.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:10.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:10.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:10.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:24:10.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:24:10.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-dup/0 2021-05-12T12:24:10.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:24:10.436 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:24:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=50b62a80-9348-40b4-a4b9-760380b62594 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148024/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:10.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-dup/0/whoami 2021-05-12T12:24:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:24:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:24:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:24:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:24:10.451 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:10.448+0000 7fa349fe20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:10.453 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:10.450+0000 7fa349fe20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:10.455 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:10.452+0000 7fa349fe20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:10.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:24:10.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:10.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:24:10.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:10.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:10.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:10.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:10.793 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:24:10.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:10.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:10.971 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:10.969+0000 7fa349fe20c0 -1 Falling back to public interface 2021-05-12T12:24:11.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:12.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:12.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:12.178 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:24:12.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:12.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:12.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:12.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:13.540 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:13.537+0000 7fa349fe20c0 -1 osd.0 22 log_to_monitors {default=true} 2021-05-12T12:24:13.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:13.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:13.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:13.544 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:24:13.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:13.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:13.925 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 26 up_thru 17 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/150089,v1:127.0.0.1:6801/150089] [v2:127.0.0.1:6802/150089,v1:127.0.0.1:6803/150089] exists,up a9a088d3-b068-4596-b50a-7a4da8bcd5ad 2021-05-12T12:24:13.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:13.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:13.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:13.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2021-05-12T12:24:13.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2021-05-12T12:24:14.542 INFO:tasks.workunit.client.0.smithi090.stdout:3 osds: 3 up (since 0.763576s), 3 in (since 35s); epoch: e27 2021-05-12T12:24:14.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2021-05-12T12:24:14.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2021-05-12T12:24:14.929 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T12:24:14.930 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T12:24:14.930 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T12:24:14.931 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T12:24:14.931 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2021-05-12T12:24:14.931 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T12:24:14.932 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T12:24:14.932 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T12:24:14.932 INFO:tasks.workunit.client.0.smithi090.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T12:24:14.933 INFO:tasks.workunit.client.0.smithi090.stdout: "osd_objectstore": "bluestore", 2021-05-12T12:24:14.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2021-05-12T12:24:15.287 INFO:tasks.workunit.client.0.smithi090.stderr:instructed osd(s) 0 to scrub 2021-05-12T12:24:15.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2021-05-12T12:24:30.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2021-05-12T12:24:30.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:24:30.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:24:30.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:24:30.688 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:24:30.688 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:24:30.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:24:30.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:30.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:24:30.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=111669149702 2021-05-12T12:24:30.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 111669149702 2021-05-12T12:24:30.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702' 2021-05-12T12:24:30.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:30.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:24:30.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672973 2021-05-12T12:24:30.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672973 2021-05-12T12:24:30.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702 1-42949672973' 2021-05-12T12:24:30.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:30.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:24:31.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509453 2021-05-12T12:24:31.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509453 2021-05-12T12:24:31.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149702 1-42949672973 2-64424509453' 2021-05-12T12:24:31.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:31.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-111669149702 2021-05-12T12:24:31.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:31.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:24:31.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-111669149702 2021-05-12T12:24:31.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:31.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=111669149702 2021-05-12T12:24:31.122 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 111669149702 2021-05-12T12:24:31.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2021-05-12T12:24:31.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:31.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149700 -lt 111669149702 2021-05-12T12:24:31.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:24:32.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:24:32.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:32.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149702 -lt 111669149702 2021-05-12T12:24:32.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:32.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2021-05-12T12:24:32.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:32.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:24:32.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672973 2021-05-12T12:24:32.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:32.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672973 2021-05-12T12:24:32.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-05-12T12:24:32.876 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672973 2021-05-12T12:24:32.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:24:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672973 -lt 42949672973 2021-05-12T12:24:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:33.258 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-05-12T12:24:33.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:33.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:24:33.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509453 2021-05-12T12:24:33.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:33.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509453 2021-05-12T12:24:33.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-05-12T12:24:33.262 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509453 2021-05-12T12:24:33.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:24:33.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509454 -lt 64424509453 2021-05-12T12:24:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2021-05-12T12:24:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:24:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:24:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:24:33.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:24:33.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:24:33.637 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:24:33.637 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:24:33.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:24:33.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:24:33.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:24:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:24:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:24:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:24:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:24:33.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:24:34.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:24:34.077 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:24:34.077 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:24:34.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:24:34.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:34.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:24:34.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=111669149705 2021-05-12T12:24:34.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 111669149705 2021-05-12T12:24:34.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705' 2021-05-12T12:24:34.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:34.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:24:34.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672976 2021-05-12T12:24:34.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672976 2021-05-12T12:24:34.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705 1-42949672976' 2021-05-12T12:24:34.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:24:34.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:24:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509456 2021-05-12T12:24:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509456 2021-05-12T12:24:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-111669149705 1-42949672976 2-64424509456' 2021-05-12T12:24:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:34.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-111669149705 2021-05-12T12:24:34.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:34.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:24:34.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-111669149705 2021-05-12T12:24:34.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:34.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=111669149705 2021-05-12T12:24:34.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2021-05-12T12:24:34.476 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 111669149705 2021-05-12T12:24:34.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:24:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 111669149705 -lt 111669149705 2021-05-12T12:24:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2021-05-12T12:24:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:34.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:24:34.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672976 2021-05-12T12:24:34.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672976 2021-05-12T12:24:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-05-12T12:24:34.848 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672976 2021-05-12T12:24:34.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:24:35.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672976 -lt 42949672976 2021-05-12T12:24:35.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:24:35.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-05-12T12:24:35.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:24:35.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:24:35.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509456 2021-05-12T12:24:35.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:24:35.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509456 2021-05-12T12:24:35.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-05-12T12:24:35.226 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 64424509456 2021-05-12T12:24:35.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:24:35.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509456 -lt 64424509456 2021-05-12T12:24:35.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:24:35.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:24:35.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:24:36.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 16 == 0 2021-05-12T12:24:36.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:24:36.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:24:36.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:24:36.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:24:36.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:24:36.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:24:36.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:24:36.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=16 2021-05-12T12:24:36.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:24:36.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:24:36.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:24:36.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 16 = 16 2021-05-12T12:24:36.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:24:36.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:24:36.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2021-05-12T12:24:36.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:24:36.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:36.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:24:36.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:36.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:36.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:36.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:36.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:37.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:37.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:37.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:37.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:37.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:37.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:37.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:37.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:37.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:37.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:37.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:37.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:24:37.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:37.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:24:37.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148024 2021-05-12T12:24:37.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:37.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:37.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:24:37.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-dup 0 2021-05-12T12:24:37.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T12:24:37.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:24:37.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T12:24:37.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:37.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:37.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:37.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:37.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:37.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:37.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:37.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:37.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:37.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:37.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:37.043 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:37.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:37.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:37.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:24:37.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T12:24:37.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:37.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148024 2021-05-12T12:24:37.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148024 2021-05-12T12:24:37.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:37.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:37.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:24:37.049 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:24:37.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:24:37.112 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2021-05-12T12:24:37.113 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2021-05-12T12:24:37.172 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:24:37.180 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:24:37.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:24:37.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:24:37.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:24:37.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:24:37.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:24:37.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:24:37.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:24:37.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:24:37.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-fast-mark-down 2021-05-12T12:24:37.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2021-05-12T12:24:37.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2021-05-12T12:24:37.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2021-05-12T12:24:37.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2021-05-12T12:24:37.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2021-05-12T12:24:37.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2021-05-12T12:24:37.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none ' 2021-05-12T12:24:37.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none ' 2021-05-12T12:24:37.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2021-05-12T12:24:37.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2021-05-12T12:24:37.185 INFO:tasks.workunit.client.0.smithi090.stdout:Ensuring old behavior is there... 2021-05-12T12:24:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T12:24:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T12:24:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T12:24:37.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:37.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:24:37.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:37.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:37.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:37.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:37.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:37.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:37.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:37.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:37.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:37.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:37.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:37.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:37.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:37.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:37.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:37.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:37.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:24:37.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:37.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:37.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151369 2021-05-12T12:24:37.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:37.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:37.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T12:24:37.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:24:37.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:37.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.151369 2021-05-12T12:24:37.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:24:37.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:24:37.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:24:37.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:24:37.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T12:24:37.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T12:24:37.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:24:37.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:24:37.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:24:37.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T12:24:37.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T12:24:37.210 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.208+0000 7fc68f2b3700 -1 failed for service _ceph-mon._tcp 2021-05-12T12:24:37.211 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.208+0000 7fc68f2b3700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T12:24:37.211 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T12:24:37.211 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.208+0000 7fc68f2b3700 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2021-05-12T12:24:37.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:24:37.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:37.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:37.544 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:37.544 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.544 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:37.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:37.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T12:24:37.784 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.781+0000 7f109c738700 -1 failed for service _ceph-mon._tcp 2021-05-12T12:24:37.785 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.781+0000 7f109c738700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T12:24:37.785 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T12:24:37.785 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:37.781+0000 7f109c738700 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2021-05-12T12:24:37.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2021-05-12T12:24:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2021-05-12T12:24:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T12:24:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T12:24:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2021-05-12T12:24:37.790 INFO:tasks.workunit.client.0.smithi090.stdout:Testing async msgr... 2021-05-12T12:24:37.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T12:24:37.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T12:24:37.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T12:24:37.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:24:37.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:24:37.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:24:37.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:24:37.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:24:37.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:24:37.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:24:37.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:24:37.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:24:37.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:24:37.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:24:37.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:24:37.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:24:37.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:24:37.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:24:37.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:24:37.799 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:24:37.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:24:37.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:24:37.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:24:37.799 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:24:37.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:37.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151369 2021-05-12T12:24:37.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:24:37.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:24:37.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T12:24:37.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:24:37.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:37.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:37.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.151369 2021-05-12T12:24:37.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:24:37.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:24:37.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:24:37.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T12:24:37.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T12:24:37.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T12:24:37.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:24:37.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:24:37.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:24:37.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T12:24:37.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T12:24:38.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:24:38.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:38.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:38.150 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:38.150 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:38.150 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:38.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:38.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T12:24:38.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:24:38.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:24:38.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:24:38.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:24:38.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:24:38.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:24:38.413 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:24:38.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:24:38.414 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:24:38.415 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:24:38.415 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:38.415 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:38.416 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151369/ceph-mon.a.asok 2021-05-12T12:24:38.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:24:38.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.151369/ceph-mon.a.asok config get fsid 2021-05-12T12:24:38.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:24:38.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:24:38.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:24:38.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:24:38.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:24:38.510 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:24:38.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:24:38.511 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:24:38.511 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:24:38.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:38.512 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:38.513 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151369/ceph-mon.a.asok 2021-05-12T12:24:38.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:24:38.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.151369/ceph-mon.a.asok config get mon_host 2021-05-12T12:24:38.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2021-05-12T12:24:38.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2021-05-12T12:24:38.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:24:38.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:24:38.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:24:38.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2021-05-12T12:24:38.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:24:39.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:24:39.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:39.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:39.505 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:39.505 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:39.505 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:39.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:39.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:24:39.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:24:39.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:39.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2021-05-12T12:24:39.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:39.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:39.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:24:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2021-05-12T12:24:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2021-05-12T12:24:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2021-05-12T12:24:39.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:39.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:39.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:39.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:39.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:39.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:39.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:39.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:39.536 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:39.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:39.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:39.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:39.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:39.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:39.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:39.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:39.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:39.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:39.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:39.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:39.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:39.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/0 2021-05-12T12:24:39.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:39.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=6c430a29-3dd7-4f06-9384-d7d3f2b1d807 2021-05-12T12:24:39.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 6c430a29-3dd7-4f06-9384-d7d3f2b1d807' 2021-05-12T12:24:39.540 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 6c430a29-3dd7-4f06-9384-d7d3f2b1d807 2021-05-12T12:24:39.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:39.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAHyZtgkTNuIBAAs4zMCzNvJiqQ6kxBhCApxQ== 2021-05-12T12:24:39.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAHyZtgkTNuIBAAs4zMCzNvJiqQ6kxBhCApxQ=="}' 2021-05-12T12:24:39.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 6c430a29-3dd7-4f06-9384-d7d3f2b1d807 -i td/osd-fast-mark-down/0/new.json 2021-05-12T12:24:39.859 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:24:39.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/0/new.json 2021-05-12T12:24:39.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHyZtgkTNuIBAAs4zMCzNvJiqQ6kxBhCApxQ== --osd-uuid 6c430a29-3dd7-4f06-9384-d7d3f2b1d807 2021-05-12T12:24:39.899 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.891+0000 7fc6fd2390c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:39.899 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.893+0000 7fc6fd2390c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:39.900 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.895+0000 7fc6fd2390c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:39.900 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.895+0000 7fc6fd2390c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:39.900 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.895+0000 7fc6fd2390c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:39.901 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.895+0000 7fc6fd2390c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T12:24:39.942 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:39.938+0000 7fc6fd2390c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2021-05-12T12:24:44.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2021-05-12T12:24:44.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:24:44.636 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:24:44.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:24:44.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:24:45.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:24:45.111 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:24:45.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:45.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:24:45.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:24:45.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:24:45.132 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:45.128+0000 7f46b53400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:45.130+0000 7f46b53400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.136 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:45.132+0000 7f46b53400c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:45.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:24:45.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:45.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:24:45.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:45.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:45.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:45.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:45.481 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:24:45.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:45.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:45.653 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:45.649+0000 7f46b53400c0 -1 Falling back to public interface 2021-05-12T12:24:45.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:46.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:46.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:46.846 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:24:46.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:46.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:46.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:47.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:47.860 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:47.855+0000 7f46b53400c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:24:48.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:48.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:48.219 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:24:48.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:48.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:48.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:24:48.631 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/152554,v1:127.0.0.1:6803/152554] [v2:127.0.0.1:6804/152554,v1:127.0.0.1:6805/152554] exists,up 6c430a29-3dd7-4f06-9384-d7d3f2b1d807 2021-05-12T12:24:48.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:48.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:48.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:48.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2021-05-12T12:24:48.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=152554 2021-05-12T12:24:48.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:48.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2021-05-12T12:24:48.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:48.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:48.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:24:48.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:48.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2021-05-12T12:24:48.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:48.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:48.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:48.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:48.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2021-05-12T12:24:48.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2021-05-12T12:24:48.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:48.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:48.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:48.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:48.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:48.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:48.642 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:48.642 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:48.642 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:48.642 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:48.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:48.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:48.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:48.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:48.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:48.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:48.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:48.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:48.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:48.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:48.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:48.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/1 2021-05-12T12:24:48.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:48.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f51e9316-6cd9-465c-9272-8df2e5d15858 2021-05-12T12:24:48.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f51e9316-6cd9-465c-9272-8df2e5d15858' 2021-05-12T12:24:48.647 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 f51e9316-6cd9-465c-9272-8df2e5d15858 2021-05-12T12:24:48.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:48.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAQyZtgVl6XJhAAM8OXEjsisYCE4bpRgXleCg== 2021-05-12T12:24:48.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAQyZtgVl6XJhAAM8OXEjsisYCE4bpRgXleCg=="}' 2021-05-12T12:24:48.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f51e9316-6cd9-465c-9272-8df2e5d15858 -i td/osd-fast-mark-down/1/new.json 2021-05-12T12:24:49.100 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:24:49.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/1/new.json 2021-05-12T12:24:49.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAQyZtgVl6XJhAAM8OXEjsisYCE4bpRgXleCg== --osd-uuid f51e9316-6cd9-465c-9272-8df2e5d15858 2021-05-12T12:24:49.136 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.133+0000 7fa08c9c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:49.138 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.134+0000 7fa08c9c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:49.139 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.135+0000 7fa08c9c20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.136+0000 7fa08c9c20c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.136+0000 7fa08c9c20c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.136+0000 7fa08c9c20c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T12:24:49.172 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:49.168+0000 7fa08c9c20c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2021-05-12T12:24:53.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2021-05-12T12:24:53.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:24:53.962 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:24:53.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:24:53.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:24:54.452 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:24:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:24:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:24:54.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:24:54.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:24:54.475 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:54.471+0000 7f4795f920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:54.477 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:54.473+0000 7f4795f920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:54.478 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:54.474+0000 7f4795f920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:54.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:24:54.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:24:54.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:24:54.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:24:54.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:24:54.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:54.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:24:54.817 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:24:54.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:54.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:54.999 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:54.995+0000 7f4795f920c0 -1 Falling back to public interface 2021-05-12T12:24:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:56.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:56.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:56.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:24:56.176 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:24:56.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:56.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:56.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:24:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:57.200+0000 7f4795f920c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:24:57.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:24:57.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:24:57.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:24:57.543 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:24:57.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:24:57.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:24:57.928 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/153940,v1:127.0.0.1:6811/153940] [v2:127.0.0.1:6812/153940,v1:127.0.0.1:6813/153940] exists,up f51e9316-6cd9-465c-9272-8df2e5d15858 2021-05-12T12:24:57.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:24:57.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:24:57.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:24:57.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2021-05-12T12:24:57.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=153940 2021-05-12T12:24:57.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T12:24:57.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2021-05-12T12:24:57.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T12:24:57.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:24:57.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:24:57.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:24:57.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2021-05-12T12:24:57.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T12:24:57.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:24:57.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:24:57.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:24:57.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2021-05-12T12:24:57.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2021-05-12T12:24:57.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:24:57.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:24:57.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T12:24:57.938 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:24:57.938 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:24:57.938 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:24:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:24:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:24:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:24:57.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' 2021-05-12T12:24:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:24:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:24:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:24:57.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T12:24:57.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T12:24:57.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:24:57.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:24:57.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:24:57.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:24:57.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:24:57.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/2 2021-05-12T12:24:57.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:24:57.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=28b1101d-4ca3-4075-bb38-dc38b16fa862 2021-05-12T12:24:57.943 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 28b1101d-4ca3-4075-bb38-dc38b16fa862 2021-05-12T12:24:57.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 28b1101d-4ca3-4075-bb38-dc38b16fa862' 2021-05-12T12:24:57.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:24:57.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAZyZtgNK1tOBAA7AIUUNjqVyhoeu3aRo/QHw== 2021-05-12T12:24:57.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAZyZtgNK1tOBAA7AIUUNjqVyhoeu3aRo/QHw=="}' 2021-05-12T12:24:57.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 28b1101d-4ca3-4075-bb38-dc38b16fa862 -i td/osd-fast-mark-down/2/new.json 2021-05-12T12:24:58.486 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:24:58.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/2/new.json 2021-05-12T12:24:58.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZyZtgNK1tOBAA7AIUUNjqVyhoeu3aRo/QHw== --osd-uuid 28b1101d-4ca3-4075-bb38-dc38b16fa862 2021-05-12T12:24:58.523 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.520+0000 7fc1fe66f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:58.536 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.533+0000 7fc1fe66f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:58.537 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.534+0000 7fc1fe66f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:24:58.538 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.535+0000 7fc1fe66f0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:58.538 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.535+0000 7fc1fe66f0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:58.538 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.535+0000 7fc1fe66f0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T12:24:58.649 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:24:58.646+0000 7fc1fe66f0c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2021-05-12T12:25:03.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2021-05-12T12:25:03.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:03.194 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:25:03.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:25:03.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:03.677 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:25:03.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:25:03.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:03.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=3105c981-96e6-48c4-ad55-645f1760ada3 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151369/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:03.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:03.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:03.700 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:03.697+0000 7fb3d4c010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:03.701 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:03.699+0000 7fb3d4c010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:03.703 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:03.700+0000 7fb3d4c010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:04.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:25:04.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:25:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:04.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:04.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:04.047 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:25:04.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:04.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:04.217 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:04.215+0000 7fb3d4c010c0 -1 Falling back to public interface 2021-05-12T12:25:04.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:05.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:05.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:05.431 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:25:05.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:05.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:05.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:05.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:06.461 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:06.458+0000 7fb3d4c010c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:25:06.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:06.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:06.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:06.794 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:25:06.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:06.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:07.159 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/155334,v1:127.0.0.1:6819/155334] [v2:127.0.0.1:6820/155334,v1:127.0.0.1:6821/155334] exists,up 28b1101d-4ca3-4075-bb38-dc38b16fa862 2021-05-12T12:25:07.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:07.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:07.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:07.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2021-05-12T12:25:07.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=155334 2021-05-12T12:25:07.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2021-05-12T12:25:07.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:25:07.562 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' does not exist 2021-05-12T12:25:07.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:25:07.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:25:08.384 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' already exists 2021-05-12T12:25:08.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:25:09.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:25:10.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2021-05-12T12:25:10.630 INFO:tasks.workunit.client.0.smithi090.stdout:hints = 1 2021-05-12T12:25:10.630 INFO:tasks.workunit.client.0.smithi090.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2021-05-12T12:25:10.630 INFO:tasks.workunit.client.0.smithi090.stdout:Object prefix: benchmark_data_smithi090_155944 2021-05-12T12:25:10.630 INFO:tasks.workunit.client.0.smithi090.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T12:25:10.631 INFO:tasks.workunit.client.0.smithi090.stdout: 0 0 0 0 0 0 - 0 2021-05-12T12:25:11.630 INFO:tasks.workunit.client.0.smithi090.stdout: 1 16 65 49 0.191387 0.191406 0.242434 0.287334 2021-05-12T12:25:12.630 INFO:tasks.workunit.client.0.smithi090.stdout: 2 16 128 112 0.218722 0.246094 0.226528 0.277109 2021-05-12T12:25:13.630 INFO:tasks.workunit.client.0.smithi090.stdout:Total time run: 2.22471 2021-05-12T12:25:13.630 INFO:tasks.workunit.client.0.smithi090.stdout:Total writes made: 128 2021-05-12T12:25:13.631 INFO:tasks.workunit.client.0.smithi090.stdout:Write size: 4096 2021-05-12T12:25:13.631 INFO:tasks.workunit.client.0.smithi090.stdout:Object size: 4096 2021-05-12T12:25:13.631 INFO:tasks.workunit.client.0.smithi090.stdout:Bandwidth (MB/sec): 0.224748 2021-05-12T12:25:13.631 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Bandwidth: 0.0386699 2021-05-12T12:25:13.632 INFO:tasks.workunit.client.0.smithi090.stdout:Max bandwidth (MB/sec): 0.246094 2021-05-12T12:25:13.632 INFO:tasks.workunit.client.0.smithi090.stdout:Min bandwidth (MB/sec): 0.191406 2021-05-12T12:25:13.632 INFO:tasks.workunit.client.0.smithi090.stdout:Average IOPS: 57 2021-05-12T12:25:13.632 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev IOPS: 9.89949 2021-05-12T12:25:13.633 INFO:tasks.workunit.client.0.smithi090.stdout:Max IOPS: 63 2021-05-12T12:25:13.633 INFO:tasks.workunit.client.0.smithi090.stdout:Min IOPS: 49 2021-05-12T12:25:13.633 INFO:tasks.workunit.client.0.smithi090.stdout:Average Latency(s): 0.275654 2021-05-12T12:25:13.633 INFO:tasks.workunit.client.0.smithi090.stdout:Stddev Latency(s): 0.0707521 2021-05-12T12:25:13.633 INFO:tasks.workunit.client.0.smithi090.stdout:Max latency(s): 0.444075 2021-05-12T12:25:13.634 INFO:tasks.workunit.client.0.smithi090.stdout:Min latency(s): 0.121626 2021-05-12T12:25:13.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2021-05-12T12:25:14.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2021-05-12T12:25:14.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2021-05-12T12:25:14.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T12:25:14.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2021-05-12T12:25:14.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=153940 2021-05-12T12:25:14.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 153940 -eq 0 ']' 2021-05-12T12:25:14.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=153940 2021-05-12T12:25:14.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 153940 2021-05-12T12:25:14.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T12:25:14.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T12:25:14.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T12:25:14.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T12:25:15.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T12:25:15.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T12:25:15.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T12:25:15.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T12:25:15.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T12:25:16.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2021-05-12T12:25:16.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T12:25:16.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2021-05-12T12:25:16.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T12:25:16.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T12:25:16.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T12:25:16.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 153940 -eq 153940 ']' 2021-05-12T12:25:16.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=152554 2021-05-12T12:25:16.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 152554 -eq 153940 ']' 2021-05-12T12:25:16.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=152554 2021-05-12T12:25:16.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 152554 2021-05-12T12:25:16.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T12:25:16.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T12:25:16.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T12:25:16.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T12:25:17.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T12:25:17.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T12:25:17.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T12:25:17.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T12:25:17.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T12:25:17.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2021-05-12T12:25:17.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T12:25:17.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2021-05-12T12:25:17.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T12:25:17.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T12:25:17.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2021-05-12T12:25:17.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2021-05-12T12:25:17.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:25:17.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:25:17.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:25:17.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:17.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:17.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:17.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:17.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:17.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:17.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:17.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:17.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:17.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:17.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:17.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:17.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:17.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:17.609 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:17.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:25:17.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:25:17.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:17.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:17.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:25:17.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151369 2021-05-12T12:25:17.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:17.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:17.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2021-05-12T12:25:17.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:25:17.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-fast-mark-down 0 2021-05-12T12:25:17.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T12:25:17.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:25:17.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T12:25:17.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:17.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:17.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:17.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:17.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:17.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:17.623 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:17.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:17.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:17.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:17.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:17.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:17.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:17.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:17.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:17.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:25:17.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T12:25:17.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:17.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:17.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151369 2021-05-12T12:25:17.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151369 2021-05-12T12:25:17.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:17.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:17.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:25:17.630 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:25:17.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:25:17.692 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2021-05-12T12:25:17.693 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2021-05-12T12:25:17.752 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:25:17.760 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:25:17.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:25:17.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:25:17.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:25:17.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:25:17.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:25:17.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:25:17.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:25:17.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:25:17.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-force-create-pg 2021-05-12T12:25:17.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2021-05-12T12:25:17.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2021-05-12T12:25:17.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2021-05-12T12:25:17.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2021-05-12T12:25:17.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2021-05-12T12:25:17.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2021-05-12T12:25:17.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none ' 2021-05-12T12:25:17.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:17.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2021-05-12T12:25:17.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:25:17.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2021-05-12T12:25:17.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2021-05-12T12:25:17.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: TEST_reuse_id td/osd-force-create-pg 2021-05-12T12:25:17.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:20: TEST_reuse_id: local dir=td/osd-force-create-pg 2021-05-12T12:25:17.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: setup td/osd-force-create-pg 2021-05-12T12:25:17.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2021-05-12T12:25:17.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2021-05-12T12:25:17.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T12:25:17.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:25:17.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T12:25:17.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:17.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:17.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:17.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:17.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:17.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:17.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:25:17.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:25:17.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:25:17.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:25:17.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:25:17.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:25:17.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:25:17.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:25:17.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:25:17.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:25:17.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:25:17.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T12:25:17.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:25:17.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:17.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:17.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156106 2021-05-12T12:25:17.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:25:17.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:25:17.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2021-05-12T12:25:17.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:25:17.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:17.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:17.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.156106 2021-05-12T12:25:17.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:25:17.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:25:17.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:25:17.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:25:17.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2021-05-12T12:25:17.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:23: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2021-05-12T12:25:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:25:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:25:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:25:17.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2021-05-12T12:25:17.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:18.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:25:18.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:18.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:18.165 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:18.166 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:18.166 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:18.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:18.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T12:25:18.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:25:18.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:25:18.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:25:18.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:25:18.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:25:18.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:25:18.419 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:25:18.419 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:25:18.419 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:25:18.420 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:25:18.420 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:18.420 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:18.420 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156106/ceph-mon.a.asok 2021-05-12T12:25:18.420 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:25:18.421 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156106/ceph-mon.a.asok config get fsid 2021-05-12T12:25:18.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:25:18.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:25:18.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:25:18.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:25:18.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:25:18.514 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:25:18.515 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:25:18.515 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:25:18.515 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:25:18.515 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:18.515 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:18.516 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156106/ceph-mon.a.asok 2021-05-12T12:25:18.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:25:18.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156106/ceph-mon.a.asok config get mon_host 2021-05-12T12:25:18.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2021-05-12T12:25:18.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2021-05-12T12:25:18.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:25:18.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:25:18.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:25:18.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2021-05-12T12:25:18.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:25:19.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:25:19.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:19.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:19.470 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:19.471 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:19.471 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:19.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:19.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:25:19.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:25:19.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2021-05-12T12:25:19.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:19.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:19.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:25:19.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:19.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T12:25:19.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:19.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:19.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:19.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:19.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T12:25:19.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T12:25:19.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:19.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:19.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:19.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:19.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:19.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:19.498 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:19.499 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:19.499 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:19.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:19.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:19.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:19.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:19.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:19.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:19.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:19.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:19.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:19.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:19.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:19.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:19.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T12:25:19.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:19.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0a24a325-8e5e-4904-b926-8bfc6988d81e 2021-05-12T12:25:19.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 0a24a325-8e5e-4904-b926-8bfc6988d81e' 2021-05-12T12:25:19.505 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 0a24a325-8e5e-4904-b926-8bfc6988d81e 2021-05-12T12:25:19.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:19.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAvyZtg6hEMHhAAKr6+kfw+2rifOiFBAWIMdA== 2021-05-12T12:25:19.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAvyZtg6hEMHhAAKr6+kfw+2rifOiFBAWIMdA=="}' 2021-05-12T12:25:19.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0a24a325-8e5e-4904-b926-8bfc6988d81e -i td/osd-force-create-pg/0/new.json 2021-05-12T12:25:19.851 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:25:19.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/0/new.json 2021-05-12T12:25:19.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAvyZtg6hEMHhAAKr6+kfw+2rifOiFBAWIMdA== --osd-uuid 0a24a325-8e5e-4904-b926-8bfc6988d81e 2021-05-12T12:25:19.883 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.880+0000 7fdbd8ef90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:19.884 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.882+0000 7fdbd8ef90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:19.886 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.883+0000 7fdbd8ef90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:19.887 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.884+0000 7fdbd8ef90c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:19.887 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.884+0000 7fdbd8ef90c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:19.887 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.884+0000 7fdbd8ef90c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T12:25:19.932 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:19.929+0000 7fdbd8ef90c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2021-05-12T12:25:24.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2021-05-12T12:25:24.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:24.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:25:24.576 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:25:24.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:25.045 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:25:25.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:25:25.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:25.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:25.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:25.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:25.064 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:25.061+0000 7f81f3efc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.066 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:25.063+0000 7f81f3efc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.068 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:25.065+0000 7f81f3efc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:25.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:25:25.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:25.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:25:25.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:25.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:25.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:25.399 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:25:25.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:25.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:25.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:25.582 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:25.579+0000 7f81f3efc0c0 -1 Falling back to public interface 2021-05-12T12:25:25.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:26.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:26.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:26.754 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:25:26.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:27.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:27.783 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:27.780+0000 7f81f3efc0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:25:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:28.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:28.135 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:25:28.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:28.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:28.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:25:28.529 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/157226,v1:127.0.0.1:6803/157226] [v2:127.0.0.1:6804/157226,v1:127.0.0.1:6805/157226] exists,up 0a24a325-8e5e-4904-b926-8bfc6988d81e 2021-05-12T12:25:28.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:28.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:28.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:28.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2021-05-12T12:25:28.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:28.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:28.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:25:28.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:28.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T12:25:28.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:28.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:28.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:28.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:28.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T12:25:28.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T12:25:28.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:28.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:28.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:28.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:28.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:28.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:28.538 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:28.539 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:28.539 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:28.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:28.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:28.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:28.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:28.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:28.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:28.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:28.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:28.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:28.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:28.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:28.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:28.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T12:25:28.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:28.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=748a72f6-aed3-4b13-9299-7554816d7eca 2021-05-12T12:25:28.543 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 748a72f6-aed3-4b13-9299-7554816d7eca 2021-05-12T12:25:28.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 748a72f6-aed3-4b13-9299-7554816d7eca' 2021-05-12T12:25:28.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:28.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA4yZtgHtFLIBAAc7a9WrQvcsBGWYpOklY0pw== 2021-05-12T12:25:28.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA4yZtgHtFLIBAAc7a9WrQvcsBGWYpOklY0pw=="}' 2021-05-12T12:25:28.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 748a72f6-aed3-4b13-9299-7554816d7eca -i td/osd-force-create-pg/1/new.json 2021-05-12T12:25:29.064 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:25:29.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/1/new.json 2021-05-12T12:25:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA4yZtgHtFLIBAAc7a9WrQvcsBGWYpOklY0pw== --osd-uuid 748a72f6-aed3-4b13-9299-7554816d7eca 2021-05-12T12:25:29.097 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.094+0000 7f05a9f2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.114 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.111+0000 7f05a9f2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.115 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.113+0000 7f05a9f2f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:29.117 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.114+0000 7f05a9f2f0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:29.117 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.114+0000 7f05a9f2f0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:29.117 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.114+0000 7f05a9f2f0c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T12:25:29.210 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:29.207+0000 7f05a9f2f0c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2021-05-12T12:25:33.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2021-05-12T12:25:33.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:33.711 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:25:33.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:25:33.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:34.200 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:25:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:25:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:34.219 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:34.216+0000 7f4d4b3150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:34.221 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:34.218+0000 7f4d4b3150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:34.222 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:34.219+0000 7f4d4b3150c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:34.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:25:34.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:34.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:25:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:34.575 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:25:34.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:34.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:34.741 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:34.739+0000 7f4d4b3150c0 -1 Falling back to public interface 2021-05-12T12:25:34.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:35.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:35.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:35.952 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:25:35.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:35.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:35.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:36.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:36.979 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:36.976+0000 7f4d4b3150c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:25:37.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:37.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:37.392 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:25:37.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:37.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:37.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:25:37.747 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/158611,v1:127.0.0.1:6811/158611] [v2:127.0.0.1:6812/158611,v1:127.0.0.1:6813/158611] exists,up 748a72f6-aed3-4b13-9299-7554816d7eca 2021-05-12T12:25:37.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:37.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:37.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:37.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2021-05-12T12:25:37.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:37.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:25:37.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:25:37.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:25:37.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T12:25:37.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:37.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:37.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:37.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:37.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T12:25:37.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T12:25:37.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:37.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:25:37.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:37.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:25:37.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:37.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:37.756 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:37.756 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:37.757 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:37.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:37.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:37.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:37.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:25:37.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:25:37.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:37.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:37.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:37.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:37.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:37.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:25:37.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:25:37.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T12:25:37.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:25:37.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=477e0a35-d74c-45ef-be44-29a866099199 2021-05-12T12:25:37.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 477e0a35-d74c-45ef-be44-29a866099199' 2021-05-12T12:25:37.760 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 477e0a35-d74c-45ef-be44-29a866099199 2021-05-12T12:25:37.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:25:37.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBByZtgZGJpLRAAC5F1exPB+TaO2VdyGl3exg== 2021-05-12T12:25:37.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBByZtgZGJpLRAAC5F1exPB+TaO2VdyGl3exg=="}' 2021-05-12T12:25:37.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 477e0a35-d74c-45ef-be44-29a866099199 -i td/osd-force-create-pg/2/new.json 2021-05-12T12:25:38.257 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:25:38.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/2/new.json 2021-05-12T12:25:38.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBByZtgZGJpLRAAC5F1exPB+TaO2VdyGl3exg== --osd-uuid 477e0a35-d74c-45ef-be44-29a866099199 2021-05-12T12:25:38.290 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.285+0000 7f60ea12c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:38.291 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.287+0000 7f60ea12c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:38.293 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.289+0000 7f60ea12c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:38.293 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.289+0000 7f60ea12c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:38.294 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.289+0000 7f60ea12c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:38.294 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.289+0000 7f60ea12c0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T12:25:38.342 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:38.338+0000 7f60ea12c0c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2021-05-12T12:25:43.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2021-05-12T12:25:43.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:25:43.198 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:25:43.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:25:43.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:25:43.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:25:43.675 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:25:43.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:25:43.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:43.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:25:43.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:25:43.694 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:43.690+0000 7fcd8e7f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:43.696 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:43.692+0000 7fcd8e7f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:43.698 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:43.693+0000 7fcd8e7f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:44.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:25:44.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:25:44.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:25:44.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:25:44.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:25:44.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:44.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:25:44.045 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:25:44.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:44.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:44.213 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:44.209+0000 7fcd8e7f60c0 -1 Falling back to public interface 2021-05-12T12:25:44.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:45.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:45.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:45.425 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:25:45.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:25:45.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:45.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:45.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:25:46.435 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:46.431+0000 7fcd8e7f60c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:25:46.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:25:46.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:25:46.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:25:46.786 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:25:46.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:25:46.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:25:47.149 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/159996,v1:127.0.0.1:6819/159996] [v2:127.0.0.1:6820/159996,v1:127.0.0.1:6821/159996] exists,up 477e0a35-d74c-45ef-be44-29a866099199 2021-05-12T12:25:47.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:25:47.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:25:47.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:25:47.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:29: TEST_reuse_id: ceph osd pool create foo 50 2021-05-12T12:25:48.054 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'foo' already exists 2021-05-12T12:25:48.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: wait_for_clean 2021-05-12T12:25:48.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:25:48.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:25:48.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:25:48.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:25:48.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:25:48.067 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:25:48.067 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:25:48.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:25:48.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:25:48.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:25:48.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:25:48.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:25:48.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:25:48.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:25:48.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:25:48.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:25:48.549 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:25:48.549 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:25:48.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:25:48.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:48.551 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:25:48.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836486 2021-05-12T12:25:48.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836486 2021-05-12T12:25:48.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486' 2021-05-12T12:25:48.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:48.766 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:25:48.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T12:25:48.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T12:25:48.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2021-05-12T12:25:48.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:25:48.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:25:49.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542148 2021-05-12T12:25:49.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542148 2021-05-12T12:25:49.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-60129542148' 2021-05-12T12:25:49.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:49.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2021-05-12T12:25:49.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:49.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:25:49.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836486 2021-05-12T12:25:49.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:49.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836486 2021-05-12T12:25:49.069 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836486 2021-05-12T12:25:49.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-05-12T12:25:49.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:25:49.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836484 -lt 21474836486 2021-05-12T12:25:49.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:25:50.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:25:50.410 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:25:50.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836486 2021-05-12T12:25:50.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:50.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T12:25:50.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:50.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:25:50.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T12:25:50.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:50.782 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672965 2021-05-12T12:25:50.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T12:25:50.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T12:25:50.783 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:25:51.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672965 -lt 42949672965 2021-05-12T12:25:51.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:25:51.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542148 2021-05-12T12:25:51.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:25:51.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:25:51.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542148 2021-05-12T12:25:51.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:25:51.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542148 2021-05-12T12:25:51.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2021-05-12T12:25:51.164 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542148 2021-05-12T12:25:51.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:25:51.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542148 -lt 60129542148 2021-05-12T12:25:51.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:25:51.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:51.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:52.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 50 == 0 2021-05-12T12:25:52.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:52.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:52.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:52.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:52.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:52.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:52.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:52.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=28 2021-05-12T12:25:52.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:52.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:52.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:52.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 28 = 50 2021-05-12T12:25:52.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 28 '!=' -1 2021-05-12T12:25:52.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:25:52.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=28 2021-05-12T12:25:52.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:25:52.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:25:53.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:25:53.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:53.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:53.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:53.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:53.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:53.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:53.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:53.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=28 2021-05-12T12:25:53.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:53.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:53.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:53.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 28 = 50 2021-05-12T12:25:53.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 28 '!=' 28 2021-05-12T12:25:53.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:25:53.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:25:53.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:25:53.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:25:53.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:25:53.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:25:53.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:25:54.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T12:25:54.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T12:25:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 13 )) 2021-05-12T12:25:54.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:25:54.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T12:25:54.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:25:54.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:25:54.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:25:54.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:25:54.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:25:54.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:25:54.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:25:54.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:25:54.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=50 2021-05-12T12:25:54.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:25:54.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:25:54.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:25:55.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 50 = 50 2021-05-12T12:25:55.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:25:55.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:25:55.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:32: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2021-05-12T12:25:55.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:55.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:55.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:55.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:55.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:55.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2021-05-12T12:25:55.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:55.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:55.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:55.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:55.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:55.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:55.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2021-05-12T12:25:55.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:25:55.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:25:55.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:25:55.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:25:55.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:25:55.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:25:55.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2021-05-12T12:25:56.739 INFO:tasks.workunit.client.0.smithi090.stderr:PG '1.0' not found 2021-05-12T12:25:57.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2021-05-12T12:25:57.977 INFO:tasks.workunit.client.0.smithi090.stdout: marking collection for removal 2021-05-12T12:25:57.977 INFO:tasks.workunit.client.0.smithi090.stdout:setting '_remove' omap key 2021-05-12T12:25:57.978 INFO:tasks.workunit.client.0.smithi090.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T12:25:58.111 INFO:tasks.workunit.client.0.smithi090.stdout:Remove successful 2021-05-12T12:25:58.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2021-05-12T12:25:59.474 INFO:tasks.workunit.client.0.smithi090.stderr:PG '1.0' not found 2021-05-12T12:25:59.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2021-05-12T12:25:59.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:25:59.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:25:59.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:25:59.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:25:59.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T12:25:59.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:25:59.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:25:59.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:25:59.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:25:59.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T12:25:59.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T12:25:59.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:25:59.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:25:59.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:25:59.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:25:59.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:25:59.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:25:59.753 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:25:59.753 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:25:59.753 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:25:59.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:59.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:25:59.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:25:59.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:25:59.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:25:59.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:25:59.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:25:59.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:59.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:25:59.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:25:59.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T12:25:59.756 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:25:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:25:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:25:59.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/0/whoami 2021-05-12T12:25:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:25:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:25:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:25:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:25:59.769 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:59.766+0000 7f29ad1f50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:59.771 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:59.768+0000 7f29ad1f50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:25:59.772 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:25:59.770+0000 7f29ad1f50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:00.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:26:00.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:00.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:26:00.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:00.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:00.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:00.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:00.146 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:00.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:00.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:00.289 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:00.286+0000 7f29ad1f50c0 -1 Falling back to public interface 2021-05-12T12:26:00.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:01.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:01.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:01.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:01.510 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:01.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:01.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:01.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:02.544 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:02.541+0000 7f29ad1f50c0 -1 osd.0 21 log_to_monitors {default=true} 2021-05-12T12:26:02.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:02.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:02.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:02.866 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:02.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:02.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:03.249 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 25 up_thru 15 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/162291,v1:127.0.0.1:6803/162291] [v2:127.0.0.1:6804/162291,v1:127.0.0.1:6805/162291] exists,up 0a24a325-8e5e-4904-b926-8bfc6988d81e 2021-05-12T12:26:03.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:03.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:03.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:03.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2021-05-12T12:26:03.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:26:03.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:26:03.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T12:26:03.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:26:03.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T12:26:03.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:26:03.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:03.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:03.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:03.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T12:26:03.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T12:26:03.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:03.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:26:03.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:26:03.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:26:03.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:03.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:03.263 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:03.263 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:03.263 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:26:03.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:26:03.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:26:03.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:03.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:26:03.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:26:03.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:03.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:03.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:03.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:26:03.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:26:03.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T12:26:03.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T12:26:03.266 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:26:03.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:03.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/1/whoami 2021-05-12T12:26:03.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T12:26:03.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:26:03.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:26:03.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:26:03.270 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:03.267+0000 7f8da7b690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:03.284 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:03.280+0000 7f8da7b690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:03.284 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:03.281+0000 7f8da7b690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:03.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T12:26:03.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:03.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:26:03.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:03.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:03.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:03.613 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:03.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:03.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:03.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:03.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:03.797+0000 7f8da7b690c0 -1 Falling back to public interface 2021-05-12T12:26:04.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:05.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:05.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:05.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:05.006 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:05.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:05.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:05.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:06.239 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:06.236+0000 7f8da7b690c0 -1 osd.1 21 log_to_monitors {default=true} 2021-05-12T12:26:06.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:06.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:06.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:06.362 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:06.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:06.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:06.729 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 28 up_thru 16 down_at 23 last_clean_interval [10,21) [v2:127.0.0.1:6810/162848,v1:127.0.0.1:6811/162848] [v2:127.0.0.1:6812/162848,v1:127.0.0.1:6813/162848] exists,up 748a72f6-aed3-4b13-9299-7554816d7eca 2021-05-12T12:26:06.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:06.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:06.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:06.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2021-05-12T12:26:06.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T12:26:06.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:26:06.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T12:26:06.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:26:06.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T12:26:06.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T12:26:06.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:06.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:06.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:06.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T12:26:06.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T12:26:06.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:06.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:26:06.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T12:26:06.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:26:06.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:06.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:06.737 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:06.738 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:06.738 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:26:06.738 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:26:06.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' 2021-05-12T12:26:06.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T12:26:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T12:26:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:06.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:06.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:26:06.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:26:06.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T12:26:06.741 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:26:06.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T12:26:06.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=0b86c8b4-2e50-4e89-b964-190c505a7f71 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156106/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:06.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/2/whoami 2021-05-12T12:26:06.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T12:26:06.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:26:06.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:26:06.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:26:06.750 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:06.747+0000 7f4f1732d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:06.751 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:06.748+0000 7f4f1732d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:06.753 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:06.750+0000 7f4f1732d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:07.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T12:26:07.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:07.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:26:07.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:07.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:07.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:07.141 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:07.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:07.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:07.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:07.272 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:07.269+0000 7f4f1732d0c0 -1 Falling back to public interface 2021-05-12T12:26:07.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:08.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:08.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:08.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:08.516 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:08.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:08.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:08.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:09.483 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:09.480+0000 7f4f1732d0c0 -1 osd.2 21 log_to_monitors {default=true} 2021-05-12T12:26:09.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:09.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:09.899 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:09.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:09.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:09.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:10.263 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 31 up_thru 31 down_at 26 last_clean_interval [14,21) [v2:127.0.0.1:6818/163405,v1:127.0.0.1:6819/163405] [v2:127.0.0.1:6820/163405,v1:127.0.0.1:6821/163405] exists,up 477e0a35-d74c-45ef-be44-29a866099199 2021-05-12T12:26:10.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:10.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:10.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:10.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: sleep 10 2021-05-12T12:26:20.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: ceph pg ls 2021-05-12T12:26:20.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep 1.0 2021-05-12T12:26:20.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep stale 2021-05-12T12:26:20.620 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 31s 0'0 20:12 [1]p1 [1]p1 2021-05-12T12:25:47.653469+0000 2021-05-12T12:25:47.653469+0000 2021-05-12T12:26:20.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:44: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2021-05-12T12:26:21.106 INFO:tasks.workunit.client.0.smithi090.stderr:pg 1.0 already creating 2021-05-12T12:26:21.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: wait_for_clean 2021-05-12T12:26:21.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:26:21.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:26:21.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:26:21.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:26:21.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:26:21.118 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:26:21.119 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:26:21.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:26:21.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:26:21.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:26:21.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:26:21.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:26:21.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:26:21.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:26:21.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:26:21.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:26:21.633 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:26:21.633 INFO:tasks.workunit.client.0.smithi090.stderr:2' 2021-05-12T12:26:21.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:26:21.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:21.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:26:21.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=107374182406 2021-05-12T12:26:21.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 107374182406 2021-05-12T12:26:21.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406' 2021-05-12T12:26:21.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:21.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:26:21.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=120259084294 2021-05-12T12:26:21.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 120259084294 2021-05-12T12:26:21.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406 1-120259084294' 2021-05-12T12:26:21.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:26:21.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:26:22.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=133143986181 2021-05-12T12:26:22.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 133143986181 2021-05-12T12:26:22.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-107374182406 1-120259084294 2-133143986181' 2021-05-12T12:26:22.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:22.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182406 2021-05-12T12:26:22.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:22.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:26:22.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-107374182406 2021-05-12T12:26:22.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:22.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=107374182406 2021-05-12T12:26:22.009 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 107374182406 2021-05-12T12:26:22.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2021-05-12T12:26:22.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:26:22.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 107374182406 -lt 107374182406 2021-05-12T12:26:22.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:22.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-120259084294 2021-05-12T12:26:22.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:22.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:26:22.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-120259084294 2021-05-12T12:26:22.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:22.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=120259084294 2021-05-12T12:26:22.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 120259084294' 2021-05-12T12:26:22.364 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 120259084294 2021-05-12T12:26:22.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:26:22.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 120259084294 -lt 120259084294 2021-05-12T12:26:22.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:26:22.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986181 2021-05-12T12:26:22.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:26:22.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:26:22.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-133143986181 2021-05-12T12:26:22.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:26:22.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=133143986181 2021-05-12T12:26:22.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 133143986181' 2021-05-12T12:26:22.744 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 133143986181 2021-05-12T12:26:22.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:26:23.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 133143986181 -lt 133143986181 2021-05-12T12:26:23.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:26:23.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:26:23.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:26:23.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 50 == 0 2021-05-12T12:26:23.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:26:23.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:26:23.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:26:23.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:26:23.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:26:23.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:26:23.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:26:23.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=50 2021-05-12T12:26:23.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:26:23.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:26:23.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:26:24.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 50 = 50 2021-05-12T12:26:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:26:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:26:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:26:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-force-create-pg 0 2021-05-12T12:26:24.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T12:26:24.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:26:24.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T12:26:24.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:26:24.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:26:24.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:26:24.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:26:24.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:26:24.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:26:24.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:26:24.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:26:24.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:26:24.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:26:24.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:26:24.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:26:24.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:24.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:26:24.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:26:24.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:24.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:26:24.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:26:24.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T12:26:24.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:26:24.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:24.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156106 2021-05-12T12:26:24.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156106 2021-05-12T12:26:24.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:26:24.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:26:24.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:26:24.544 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:26:24.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:26:24.607 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2021-05-12T12:26:24.607 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2021-05-12T12:26:24.667 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:26:24.675 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:26:24.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:26:24.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:26:24.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:26:24.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:26:24.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:26:24.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:26:24.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:26:24.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:26:24.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-markdown 2021-05-12T12:26:24.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2021-05-12T12:26:24.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2021-05-12T12:26:24.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2021-05-12T12:26:24.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2021-05-12T12:26:24.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2021-05-12T12:26:24.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2021-05-12T12:26:24.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none ' 2021-05-12T12:26:24.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:24.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2021-05-12T12:26:24.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:26:24.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2021-05-12T12:26:24.682 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_markdown_boot_exceed_time 2021-05-12T12:26:24.682 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_markdown_exceed_maxdown_count 2021-05-12T12:26:24.682 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_osd_stop' 2021-05-12T12:26:24.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:26:24.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:26:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:26:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:26:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:26:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:26:24.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:26:24.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:26:24.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:26:24.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:26:24.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:26:24.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:26:24.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:26:24.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:26:24.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:26:24.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:26:24.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:26:24.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:26:24.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:26:24.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:24.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:26:24.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:26:24.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:26:24.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:26:24.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:26:24.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:26:24.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:26:24.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:24.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:24.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:26:24.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:26:24.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:26:24.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:26:24.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:26:24.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:24.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:24.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164437 2021-05-12T12:26:24.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:26:24.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:26:24.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:26:24.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:26:24.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:26:24.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2021-05-12T12:26:24.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2021-05-12T12:26:24.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2021-05-12T12:26:24.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:26:24.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:26:24.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:26:24.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:26:24.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:26:24.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:26:25.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:26:25.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:25.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:25.035 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:25.036 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:25.036 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:25.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:25.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:26:25.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:26:25.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:26:25.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:26:25.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:26:25.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:26:25.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:26:25.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:26:25.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:26:25.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:26:25.306 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:26:25.307 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:25.307 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:25.307 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:26:25.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:26:25.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get fsid 2021-05-12T12:26:25.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:26:25.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:26:25.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:26:25.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:26:25.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:26:25.402 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:26:25.402 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:26:25.403 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:26:25.403 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:26:25.403 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:25.404 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:25.404 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:26:25.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:26:25.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get mon_host 2021-05-12T12:26:25.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2021-05-12T12:26:25.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:26:25.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:26:25.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:26:25.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:26:25.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:26:25.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:26:26.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:26:26.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:26.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:26.358 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:26.359 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:26.359 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:26.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:26.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:26:26.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:26:26.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2021-05-12T12:26:26.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:26.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:26.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:26:26.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:26.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:26:26.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:26.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:26:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:26:26.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:26.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:26.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:26.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:26.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:26.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:26.387 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:26.387 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:26.387 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:26.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:26.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:26.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:26.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:26.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:26.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:26.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:26.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:26.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:26.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:26.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:26.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:26.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:26:26.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:26.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7543e7b1-52fc-49cf-9376-10580f8dbb3a 2021-05-12T12:26:26.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 7543e7b1-52fc-49cf-9376-10580f8dbb3a' 2021-05-12T12:26:26.391 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 7543e7b1-52fc-49cf-9376-10580f8dbb3a 2021-05-12T12:26:26.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:26.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQByyZtgkyBoFxAAtsUf0VSmv2JOLLPJuRoDqw== 2021-05-12T12:26:26.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQByyZtgkyBoFxAAtsUf0VSmv2JOLLPJuRoDqw=="}' 2021-05-12T12:26:26.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7543e7b1-52fc-49cf-9376-10580f8dbb3a -i td/osd-markdown/0/new.json 2021-05-12T12:26:26.712 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:26.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:26:26.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQByyZtgkyBoFxAAtsUf0VSmv2JOLLPJuRoDqw== --osd-uuid 7543e7b1-52fc-49cf-9376-10580f8dbb3a 2021-05-12T12:26:26.744 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.741+0000 7fa106e5c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.746 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.743+0000 7fa106e5c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.747 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.744+0000 7fa106e5c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:26.748 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.745+0000 7fa106e5c0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:26.748 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.745+0000 7fa106e5c0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:26.748 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.745+0000 7fa106e5c0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:26:26.787 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:26.784+0000 7fa106e5c0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:26:31.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:26:31.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:31.454 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:26:31.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:26:31.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:31.932 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:26:31.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:26:31.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:31.951 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:31.948+0000 7f67137a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:31.953 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:31.950+0000 7f67137a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:31.954 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:31.951+0000 7f67137a60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:32.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:26:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:26:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:32.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:32.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:32.286 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:32.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:32.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:32.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:32.469 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:32.467+0000 7f67137a60c0 -1 Falling back to public interface 2021-05-12T12:26:32.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:33.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:33.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:33.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:33.642 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:33.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:33.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:34.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:34.655 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:34.652+0000 7f67137a60c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:26:35.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:35.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:35.009 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:35.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:35.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:35.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:26:35.385 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165557,v1:127.0.0.1:6803/165557] [v2:127.0.0.1:6804/165557,v1:127.0.0.1:6805/165557] exists,up 7543e7b1-52fc-49cf-9376-10580f8dbb3a 2021-05-12T12:26:35.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:35.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:35.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:35.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2021-05-12T12:26:35.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:35.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:35.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:26:35.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:35.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:26:35.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:35.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:35.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:35.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:35.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:26:35.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:26:35.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:35.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:35.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:35.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:35.394 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:35.394 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:35.394 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:35.394 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:35.395 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:35.395 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:35.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:35.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:35.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:35.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:35.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:35.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:35.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:35.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:35.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:35.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:35.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:35.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:26:35.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:35.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7c0c42f9-597a-41cc-8d6f-4861344b4bb8 2021-05-12T12:26:35.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 7c0c42f9-597a-41cc-8d6f-4861344b4bb8' 2021-05-12T12:26:35.399 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 7c0c42f9-597a-41cc-8d6f-4861344b4bb8 2021-05-12T12:26:35.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:35.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB7yZtg/4isFxAABPVZZbqLN9S0r0LsY7dcyw== 2021-05-12T12:26:35.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB7yZtg/4isFxAABPVZZbqLN9S0r0LsY7dcyw=="}' 2021-05-12T12:26:35.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7c0c42f9-597a-41cc-8d6f-4861344b4bb8 -i td/osd-markdown/1/new.json 2021-05-12T12:26:35.909 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:35.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:26:35.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB7yZtg/4isFxAABPVZZbqLN9S0r0LsY7dcyw== --osd-uuid 7c0c42f9-597a-41cc-8d6f-4861344b4bb8 2021-05-12T12:26:35.942 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.939+0000 7f925b6f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:35.967 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.964+0000 7f925b6f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:35.969 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.965+0000 7f925b6f60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:35.969 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.966+0000 7f925b6f60c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:35.970 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.966+0000 7f925b6f60c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:35.970 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:35.966+0000 7f925b6f60c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:26:36.067 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:36.064+0000 7f925b6f60c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:26:40.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:26:40.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:40.831 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:26:40.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:26:40.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:41.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:26:41.310 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:26:41.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:41.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:41.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:41.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:41.328 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:41.325+0000 7f28acc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.330 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:41.326+0000 7f28acc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.332 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:41.328+0000 7f28acc1c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:41.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:26:41.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:41.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:26:41.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:41.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:41.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:41.667 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:41.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:41.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:41.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:41.847 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:41.843+0000 7f28acc1c0c0 -1 Falling back to public interface 2021-05-12T12:26:42.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:43.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:43.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:43.026 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:43.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:43.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:43.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:43.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:44.021 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:44.017+0000 7f28acc1c0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:26:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:44.388 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:44.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:44.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:44.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:26:44.805 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/166942,v1:127.0.0.1:6811/166942] [v2:127.0.0.1:6812/166942,v1:127.0.0.1:6813/166942] exists,up 7c0c42f9-597a-41cc-8d6f-4861344b4bb8 2021-05-12T12:26:44.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:44.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:44.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:44.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2021-05-12T12:26:44.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:26:44.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:26:44.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:26:44.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:26:44.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:26:44.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:26:44.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:26:44.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:26:44.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:26:44.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:26:44.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:26:44.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:26:44.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:26:44.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:26:44.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:26:44.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:26:44.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:26:44.815 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:26:44.815 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:26:44.815 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:26:44.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:44.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:26:44.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:26:44.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:26:44.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:26:44.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:26:44.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:26:44.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:26:44.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:26:44.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:44.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:26:44.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:26:44.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:26:44.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:26:44.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=392113c0-3a57-41ce-b45a-5a051330dfef 2021-05-12T12:26:44.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 392113c0-3a57-41ce-b45a-5a051330dfef' 2021-05-12T12:26:44.820 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 392113c0-3a57-41ce-b45a-5a051330dfef 2021-05-12T12:26:44.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:26:44.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCEyZtgM8vdMBAAiSBbpc8NGigIsh7+c4jAzg== 2021-05-12T12:26:44.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCEyZtgM8vdMBAAiSBbpc8NGigIsh7+c4jAzg=="}' 2021-05-12T12:26:44.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 392113c0-3a57-41ce-b45a-5a051330dfef -i td/osd-markdown/2/new.json 2021-05-12T12:26:45.276 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:45.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:26:45.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCEyZtgM8vdMBAAiSBbpc8NGigIsh7+c4jAzg== --osd-uuid 392113c0-3a57-41ce-b45a-5a051330dfef 2021-05-12T12:26:45.309 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.305+0000 7f04ceafd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:45.311 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.307+0000 7f04ceafd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:45.311 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.308+0000 7f04ceafd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:45.312 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.308+0000 7f04ceafd0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:45.312 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.308+0000 7f04ceafd0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:45.313 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.308+0000 7f04ceafd0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:26:45.355 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:45.352+0000 7f04ceafd0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:26:50.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:26:50.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:26:50.119 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:26:50.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:26:50.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:26:50.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:26:50.609 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:26:50.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:26:50.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:26:50.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:26:50.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:26:50.628 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:50.624+0000 7fdf3517e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:50.629 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:50.626+0000 7fdf3517e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:50.631 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:50.627+0000 7fdf3517e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:26:50.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:26:51.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:26:51.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:26:51.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:26:51.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:26:51.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:51.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:26:51.002 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:26:51.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:51.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:51.146 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:51.142+0000 7fdf3517e0c0 -1 Falling back to public interface 2021-05-12T12:26:51.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:52.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:52.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:52.322 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:26:52.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:26:52.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:52.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:52.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:26:53.317 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:26:53.313+0000 7fdf3517e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:26:53.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:26:53.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:26:53.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:26:53.680 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:26:53.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:26:53.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:26:54.063 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/168327,v1:127.0.0.1:6819/168327] [v2:127.0.0.1:6820/168327,v1:127.0.0.1:6821/168327] exists,up 392113c0-3a57-41ce-b45a-5a051330dfef 2021-05-12T12:26:54.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:26:54.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:26:54.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:26:54.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2021-05-12T12:26:54.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:26:54.490 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' does not exist 2021-05-12T12:26:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:26:54.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:26:55.209 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' already exists 2021-05-12T12:26:55.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:26:56.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:26:57.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2021-05-12T12:26:57.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2021-05-12T12:26:57.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2021-05-12T12:26:57.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:26:57.883 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:26:57.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2021-05-12T12:26:58.009 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:26:58.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2021-05-12T12:26:58.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2021-05-12T12:26:58.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2021-05-12T12:26:58.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:26:58.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2021-05-12T12:26:58.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:26:58.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:26:58.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:26:58.491 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:26:58.491 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:26:58.491 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:26:58.492 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:26:58.492 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:26:58.492 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:26:58.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:26:58.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:26:58.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:26:58.870 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:26:58.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:26:58.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:26:59.404 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:26:59.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:27:09.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:27:09.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:09.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:27:09.893 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:27:09.894 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:27:09.894 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:27:09.894 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:09.894 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:27:09.895 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:27:09.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:27:09.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:27:09.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:27:10.279 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:10.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:27:10.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:27:10.661 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:27:10.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:27:20.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:27:20.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:20.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:27:21.167 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:27:21.167 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:27:21.168 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:27:21.168 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:21.168 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:27:21.168 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:27:21.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:27:21.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:27:21.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:27:21.555 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:21.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:27:21.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:27:21.945 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:27:21.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:27:31.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2021-05-12T12:27:46.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:27:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2021-05-12T12:27:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2021-05-12T12:27:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2021-05-12T12:27:47.449 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:27:47.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:27:47.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:27:47.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:27:47.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:27:47.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:27:47.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:27:47.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:27:47.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:27:47.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:27:47.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:27:47.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:27:47.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:27:47.559 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:27:47.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:27:47.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:27:47.560 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:27:47.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:47.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:27:47.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:27:47.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:47.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:27:47.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:27:47.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:27:47.569 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:27:47.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:47.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:47.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:27:47.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:27:47.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:27:47.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:27:47.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:27:47.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:27:47.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:27:47.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:27:47.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:27:47.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:27:47.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:27:47.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:27:47.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:27:47.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:27:47.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:27:47.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:27:47.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:27:47.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:27:47.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:27:47.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:27:47.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:27:47.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:27:47.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:47.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:27:47.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:27:47.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:27:47.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:27:47.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:27:47.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:27:47.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:27:47.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:47.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:47.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:27:47.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:27:47.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:27:47.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:27:47.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:27:47.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:47.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:47.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164437 2021-05-12T12:27:47.587 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:27:47.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:27:47.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:27:47.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:27:47.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2021-05-12T12:27:47.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2021-05-12T12:27:47.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2021-05-12T12:27:47.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:27:47.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:27:47.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:27:47.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:27:47.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:27:47.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:27:47.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:27:47.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:47.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:47.971 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:47.972 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:47.972 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:47.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:47.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:27:48.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:27:48.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:27:48.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:27:48.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:27:48.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:27:48.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:27:48.211 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:27:48.211 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:27:48.212 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:27:48.212 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:27:48.213 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:48.213 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:48.213 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:27:48.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:27:48.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get fsid 2021-05-12T12:27:48.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:27:48.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:27:48.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:27:48.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:27:48.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:27:48.305 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:27:48.305 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:27:48.306 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:27:48.306 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:27:48.307 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:48.307 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:48.307 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:27:48.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:27:48.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get mon_host 2021-05-12T12:27:48.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2021-05-12T12:27:48.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:27:48.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:27:48.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:27:48.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:27:48.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:27:48.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:27:49.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:27:49.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:49.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:49.268 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:49.269 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:49.269 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:49.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:49.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:27:49.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:27:49.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2021-05-12T12:27:49.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:27:49.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:27:49.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:27:49.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:27:49.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:27:49.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:27:49.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:27:49.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:27:49.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:27:49.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:27:49.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:27:49.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:27:49.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:27:49.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:27:49.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:27:49.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:49.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:49.297 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:49.297 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:49.298 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:49.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:49.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:49.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:27:49.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:27:49.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:27:49.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:27:49.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:27:49.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:27:49.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:27:49.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:49.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:27:49.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:27:49.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:27:49.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:27:49.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=74abf03a-4a99-46c6-9009-f95032a91736 2021-05-12T12:27:49.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 74abf03a-4a99-46c6-9009-f95032a91736' 2021-05-12T12:27:49.302 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 74abf03a-4a99-46c6-9009-f95032a91736 2021-05-12T12:27:49.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:27:49.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDFyZtgja0FEhAAXtekFpKE+vygSVVfM6LrPA== 2021-05-12T12:27:49.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDFyZtgja0FEhAAXtekFpKE+vygSVVfM6LrPA=="}' 2021-05-12T12:27:49.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 74abf03a-4a99-46c6-9009-f95032a91736 -i td/osd-markdown/0/new.json 2021-05-12T12:27:49.612 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:27:49.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:27:49.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDFyZtgja0FEhAAXtekFpKE+vygSVVfM6LrPA== --osd-uuid 74abf03a-4a99-46c6-9009-f95032a91736 2021-05-12T12:27:49.644 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.640+0000 7fafa794a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:49.646 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.642+0000 7fafa794a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:49.648 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.644+0000 7fafa794a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:49.649 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.645+0000 7fafa794a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:49.649 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.645+0000 7fafa794a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:49.649 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.645+0000 7fafa794a0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:27:49.696 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:49.692+0000 7fafa794a0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:27:54.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:27:54.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:27:54.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:27:54.318 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:27:54.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:27:54.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:27:54.788 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:27:54.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:54.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:27:54.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:27:54.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:27:54.806 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:54.803+0000 7f67b676f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:54.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:54.805+0000 7f67b676f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:54.810 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:54.806+0000 7f67b676f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:55.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:27:55.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:27:55.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:27:55.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:27:55.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:27:55.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:55.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:27:55.147 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:27:55.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:55.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:55.326 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:55.322+0000 7f67b676f0c0 -1 Falling back to public interface 2021-05-12T12:27:55.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:56.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:56.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:56.557 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:27:56.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:27:56.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:56.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:56.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:27:57.519 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:57.516+0000 7f67b676f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:27:57.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:27:57.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:27:57.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:27:57.925 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:27:57.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:27:57.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:27:58.282 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/170487,v1:127.0.0.1:6803/170487] [v2:127.0.0.1:6804/170487,v1:127.0.0.1:6805/170487] exists,up 74abf03a-4a99-46c6-9009-f95032a91736 2021-05-12T12:27:58.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:27:58.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:27:58.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:27:58.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2021-05-12T12:27:58.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:27:58.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:27:58.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:27:58.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:27:58.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:27:58.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:27:58.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:27:58.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:27:58.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:27:58.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:27:58.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:27:58.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:27:58.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:27:58.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:27:58.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:27:58.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:27:58.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:27:58.291 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:27:58.291 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:27:58.291 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:27:58.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:58.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:27:58.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:27:58.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:27:58.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:27:58.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:27:58.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:27:58.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:27:58.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:27:58.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:27:58.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:27:58.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:27:58.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:27:58.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:27:58.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=32300077-7e25-4881-a482-28b329410aed 2021-05-12T12:27:58.295 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 32300077-7e25-4881-a482-28b329410aed 2021-05-12T12:27:58.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 32300077-7e25-4881-a482-28b329410aed' 2021-05-12T12:27:58.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:27:58.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDOyZtgfwmVERAAyUoH/hfpE0hO1QMhWexrVw== 2021-05-12T12:27:58.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDOyZtgfwmVERAAyUoH/hfpE0hO1QMhWexrVw=="}' 2021-05-12T12:27:58.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 32300077-7e25-4881-a482-28b329410aed -i td/osd-markdown/1/new.json 2021-05-12T12:27:58.852 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:27:58.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:27:58.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDOyZtgfwmVERAAyUoH/hfpE0hO1QMhWexrVw== --osd-uuid 32300077-7e25-4881-a482-28b329410aed 2021-05-12T12:27:58.885 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.882+0000 7fc9a76cd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.887 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.884+0000 7fc9a76cd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.888 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.886+0000 7fc9a76cd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:27:58.889 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.886+0000 7fc9a76cd0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:58.889 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.886+0000 7fc9a76cd0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:58.890 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.886+0000 7fc9a76cd0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:27:58.936 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:27:58.933+0000 7fc9a76cd0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:28:03.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:28:03.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:28:03.475 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:28:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:28:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:28:03.955 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:28:03.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:28:03.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:28:03.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:28:03.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:28:03.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:28:03.974 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:03.971+0000 7fb216df80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:03.976 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:03.973+0000 7fb216df80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:03.977 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:03.975+0000 7fb216df80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:04.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:28:04.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:28:04.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:28:04.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:28:04.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:28:04.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:04.343 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:28:04.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:28:04.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:04.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:28:04.494 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:04.491+0000 7fb216df80c0 -1 Falling back to public interface 2021-05-12T12:28:04.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:05.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:05.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:05.697 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:28:05.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:28:05.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:05.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:28:06.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:06.650 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:06.647+0000 7fb216df80c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:28:07.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:07.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:07.068 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:28:07.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:28:07.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:07.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:28:07.423 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/171880,v1:127.0.0.1:6811/171880] [v2:127.0.0.1:6812/171880,v1:127.0.0.1:6813/171880] exists,up 32300077-7e25-4881-a482-28b329410aed 2021-05-12T12:28:07.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:28:07.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:28:07.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:28:07.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2021-05-12T12:28:07.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:28:07.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:28:07.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:28:07.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:28:07.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:28:07.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:28:07.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:28:07.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:28:07.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:28:07.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:28:07.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:28:07.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:28:07.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:28:07.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:28:07.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:28:07.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:28:07.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:28:07.433 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:28:07.433 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:28:07.433 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:28:07.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:28:07.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:28:07.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:28:07.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:28:07.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:28:07.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:28:07.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:28:07.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:28:07.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:28:07.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:28:07.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:28:07.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:28:07.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:28:07.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:28:07.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=983598a1-167c-486c-998b-2c07533750d2 2021-05-12T12:28:07.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 983598a1-167c-486c-998b-2c07533750d2' 2021-05-12T12:28:07.437 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 983598a1-167c-486c-998b-2c07533750d2 2021-05-12T12:28:07.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:28:07.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDXyZtgqtgEGhAAtfMP+3fV4GQtipn5nyRC3A== 2021-05-12T12:28:07.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDXyZtgqtgEGhAAtfMP+3fV4GQtipn5nyRC3A=="}' 2021-05-12T12:28:07.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 983598a1-167c-486c-998b-2c07533750d2 -i td/osd-markdown/2/new.json 2021-05-12T12:28:07.939 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:28:07.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:28:07.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDXyZtgqtgEGhAAtfMP+3fV4GQtipn5nyRC3A== --osd-uuid 983598a1-167c-486c-998b-2c07533750d2 2021-05-12T12:28:07.971 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.968+0000 7fe358e530c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:07.992 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.989+0000 7fe358e530c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:07.993 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.990+0000 7fe358e530c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:07.994 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.991+0000 7fe358e530c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:28:07.994 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.991+0000 7fe358e530c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:28:07.994 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:07.991+0000 7fe358e530c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:28:08.059 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:08.056+0000 7fe358e530c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:28:12.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:28:12.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:28:12.924 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:28:12.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:28:12.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:28:13.402 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:28:13.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:28:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:28:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:28:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:28:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:28:13.421 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:13.419+0000 7fe2fd0450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:13.423 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:13.421+0000 7fe2fd0450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:13.425 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:13.422+0000 7fe2fd0450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:28:13.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:28:13.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:28:13.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:28:13.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:28:13.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:28:13.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:13.759 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:28:13.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:28:13.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:13.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:13.940 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:13.937+0000 7fe2fd0450c0 -1 Falling back to public interface 2021-05-12T12:28:14.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:15.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:15.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:15.112 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:28:15.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:28:15.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:15.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:15.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:28:16.146 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:28:16.143+0000 7fe2fd0450c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:28:16.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:28:16.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:28:16.496 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:28:16.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:28:16.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:28:16.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:28:16.876 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173265,v1:127.0.0.1:6819/173265] [v2:127.0.0.1:6820/173265,v1:127.0.0.1:6821/173265] exists,up 983598a1-167c-486c-998b-2c07533750d2 2021-05-12T12:28:16.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:28:16.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:28:16.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:28:16.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2021-05-12T12:28:16.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:28:17.293 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' does not exist 2021-05-12T12:28:17.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:28:17.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:28:17.802 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' already exists 2021-05-12T12:28:17.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:28:18.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:28:20.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2021-05-12T12:28:20.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2021-05-12T12:28:20.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2021-05-12T12:28:20.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:28:20.431 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:28:20.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2021-05-12T12:28:20.556 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:28:20.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2021-05-12T12:28:20.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T12:28:20.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2021-05-12T12:28:20.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:28:20.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T12:28:20.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:20.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:20.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:21.096 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:21.096 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:28:21.096 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:28:21.096 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:21.097 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:21.097 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:21.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:21.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:21.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:21.456 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:21.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:21.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:21.839 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:28:21.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:31.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:32.325 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:32.326 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:28:32.326 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:28:32.326 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:32.326 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:32.327 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:32.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:32.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:32.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:32.690 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:32.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:32.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:33.079 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:28:33.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:43.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:43.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:43.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:43.569 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:43.569 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:28:43.569 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:28:43.569 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:43.570 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:43.570 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:43.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:43.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:43.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:43.970 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:43.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:43.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:44.398 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:28:44.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:28:54.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:28:54.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:28:54.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:28:54.880 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:28:54.880 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:28:54.880 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:28:54.881 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:54.881 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:28:54.881 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:28:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:28:54.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:28:54.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:28:55.248 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:28:55.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:28:55.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:28:55.639 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:28:55.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:29:05.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2021-05-12T12:29:20.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:29:20.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2021-05-12T12:29:20.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2021-05-12T12:29:20.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2021-05-12T12:29:21.142 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:21.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:29:21.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:29:21.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:29:21.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:29:21.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:29:21.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:29:21.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:29:21.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:29:21.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:29:21.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:29:21.250 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:29:21.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:29:21.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:29:21.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:29:21.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:29:21.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:29:21.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:21.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:29:21.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:29:21.255 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:21.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:29:21.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:29:21.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:29:21.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:29:21.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:21.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:29:21.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:29:21.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:29:21.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:29:21.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:29:21.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:29:21.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:29:21.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:29:21.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:29:21.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:29:21.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:29:21.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:29:21.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:29:21.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:29:21.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:29:21.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:29:21.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:29:21.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:29:21.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:29:21.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:29:21.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:29:21.271 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:29:21.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:21.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:29:21.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:29:21.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:29:21.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:29:21.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:29:21.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:29:21.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:29:21.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:21.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:29:21.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:29:21.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:29:21.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:29:21.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:29:21.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:21.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164437 2021-05-12T12:29:21.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:29:21.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:29:21.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:29:21.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:29:21.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2021-05-12T12:29:21.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2021-05-12T12:29:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2021-05-12T12:29:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:29:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:29:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:29:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:29:21.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:29:21.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:29:21.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:29:21.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:21.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:21.633 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:21.634 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.634 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:21.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:21.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:29:21.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:29:21.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:29:21.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:29:21.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:29:21.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:29:21.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:29:21.928 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:29:21.929 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:29:21.929 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:29:21.929 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:29:21.929 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:21.930 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:21.930 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:29:21.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:29:21.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get fsid 2021-05-12T12:29:22.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:29:22.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:29:22.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:29:22.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:29:22.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:29:22.023 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:29:22.023 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:29:22.023 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:29:22.023 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:29:22.024 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:22.024 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:22.024 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:29:22.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:29:22.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get mon_host 2021-05-12T12:29:22.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2021-05-12T12:29:22.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:29:22.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:29:22.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:29:22.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:29:22.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:29:22.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:29:22.986 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:29:22.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:22.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:22.987 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:22.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:22.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:22.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:22.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:29:22.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:29:23.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2021-05-12T12:29:23.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:23.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:23.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:29:23.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:23.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:29:23.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:23.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:23.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:23.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:23.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:29:23.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:29:23.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:23.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:23.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:23.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:23.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:23.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:23.016 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:23.016 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:23.016 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:23.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:23.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:23.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:23.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:23.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:23.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:23.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:23.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:23.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:23.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:23.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:23.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:23.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:29:23.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:23.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9e80461e-be88-48f5-afba-d06aaa90fa57 2021-05-12T12:29:23.020 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 9e80461e-be88-48f5-afba-d06aaa90fa57 2021-05-12T12:29:23.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 9e80461e-be88-48f5-afba-d06aaa90fa57' 2021-05-12T12:29:23.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:23.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAjyptgKMpFARAAnc+oh4zmgYgBQQRatxQmEA== 2021-05-12T12:29:23.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAjyptgKMpFARAAnc+oh4zmgYgBQQRatxQmEA=="}' 2021-05-12T12:29:23.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9e80461e-be88-48f5-afba-d06aaa90fa57 -i td/osd-markdown/0/new.json 2021-05-12T12:29:23.334 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:29:23.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:29:23.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAjyptgKMpFARAAnc+oh4zmgYgBQQRatxQmEA== --osd-uuid 9e80461e-be88-48f5-afba-d06aaa90fa57 2021-05-12T12:29:23.367 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.364+0000 7ff2069550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:23.369 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.366+0000 7ff2069550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:23.370 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.367+0000 7ff2069550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:23.371 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.368+0000 7ff2069550c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:23.371 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.368+0000 7ff2069550c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:23.371 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.368+0000 7ff2069550c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:29:23.409 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:23.406+0000 7ff2069550c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:29:27.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:29:27.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:27.972 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:29:27.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:29:27.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:28.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:29:28.470 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:29:28.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:28.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:28.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:28.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:28.489 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:28.486+0000 7f12675d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:28.490 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:28.488+0000 7f12675d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:28.492 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:28.489+0000 7f12675d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:29:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:28.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:29:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:28.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:28.831 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:29:28.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:28.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:29.006 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:29.003+0000 7f12675d90c0 -1 Falling back to public interface 2021-05-12T12:29:29.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:30.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:30.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:30.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:30.209 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:29:30.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:30.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:30.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:31.164 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:31.161+0000 7f12675d90c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:29:31.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:31.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:31.571 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:29:31.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:31.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:31.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:29:31.950 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/175524,v1:127.0.0.1:6803/175524] [v2:127.0.0.1:6804/175524,v1:127.0.0.1:6805/175524] exists,up 9e80461e-be88-48f5-afba-d06aaa90fa57 2021-05-12T12:29:31.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:31.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:31.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:31.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2021-05-12T12:29:31.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:31.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:31.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:29:31.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:31.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:29:31.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:31.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:31.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:31.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:31.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:29:31.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:29:31.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:31.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:31.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:31.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:31.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:31.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:31.959 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:31.960 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:31.960 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:31.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:31.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:31.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:31.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:31.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:31.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:31.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:31.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:31.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:31.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:31.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:31.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:31.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:29:31.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:31.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bdab848c-f0ce-4957-9553-9c25246e545d 2021-05-12T12:29:31.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 bdab848c-f0ce-4957-9553-9c25246e545d' 2021-05-12T12:29:31.964 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 bdab848c-f0ce-4957-9553-9c25246e545d 2021-05-12T12:29:31.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:31.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAryptgivtnORAAxhciqTWuQWNv4j0w3ydv6g== 2021-05-12T12:29:31.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAryptgivtnORAAxhciqTWuQWNv4j0w3ydv6g=="}' 2021-05-12T12:29:31.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bdab848c-f0ce-4957-9553-9c25246e545d -i td/osd-markdown/1/new.json 2021-05-12T12:29:32.428 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:29:32.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:29:32.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAryptgivtnORAAxhciqTWuQWNv4j0w3ydv6g== --osd-uuid bdab848c-f0ce-4957-9553-9c25246e545d 2021-05-12T12:29:32.461 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.458+0000 7f08bda260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:32.463 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.460+0000 7f08bda260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:32.464 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.461+0000 7f08bda260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:32.465 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.462+0000 7f08bda260c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:32.465 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.462+0000 7f08bda260c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:32.465 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.462+0000 7f08bda260c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:29:32.507 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:32.504+0000 7f08bda260c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:29:37.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:29:37.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:37.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:29:37.292 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:29:37.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:37.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:29:37.768 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:29:37.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:37.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:37.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:37.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:37.787 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:37.783+0000 7fa2bbe950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:37.788 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:37.785+0000 7fa2bbe950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:37.789 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:37.785+0000 7fa2bbe950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:38.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:29:38.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:38.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:29:38.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:38.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:38.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:38.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:38.125 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:29:38.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:38.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:38.305 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:38.301+0000 7fa2bbe950c0 -1 Falling back to public interface 2021-05-12T12:29:38.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:39.476 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:29:39.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:39.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:39.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:39.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:40.472 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:40.468+0000 7fa2bbe950c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:29:40.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:40.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:40.844 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:29:40.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:40.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:40.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:29:41.207 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176909,v1:127.0.0.1:6811/176909] [v2:127.0.0.1:6812/176909,v1:127.0.0.1:6813/176909] exists,up bdab848c-f0ce-4957-9553-9c25246e545d 2021-05-12T12:29:41.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:41.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:41.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:41.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2021-05-12T12:29:41.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:29:41.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:29:41.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:29:41.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:29:41.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:29:41.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:29:41.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:29:41.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:29:41.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:29:41.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:29:41.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:29:41.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:29:41.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:29:41.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:29:41.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:29:41.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:29:41.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:29:41.216 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:29:41.216 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:29:41.217 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:29:41.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:41.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:29:41.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:29:41.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:29:41.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:29:41.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:29:41.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:29:41.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:29:41.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:29:41.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:41.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:29:41.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:29:41.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:29:41.220 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:29:41.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=67a0c8dd-c36d-480c-a625-fc4d701f2621 2021-05-12T12:29:41.220 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 67a0c8dd-c36d-480c-a625-fc4d701f2621 2021-05-12T12:29:41.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 67a0c8dd-c36d-480c-a625-fc4d701f2621' 2021-05-12T12:29:41.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:29:41.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA1yptg1hMhDRAAQc5OMXNjEwUEGP2ywgLONg== 2021-05-12T12:29:41.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA1yptg1hMhDRAAQc5OMXNjEwUEGP2ywgLONg=="}' 2021-05-12T12:29:41.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 67a0c8dd-c36d-480c-a625-fc4d701f2621 -i td/osd-markdown/2/new.json 2021-05-12T12:29:41.738 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:29:41.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:29:41.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA1yptg1hMhDRAAQc5OMXNjEwUEGP2ywgLONg== --osd-uuid 67a0c8dd-c36d-480c-a625-fc4d701f2621 2021-05-12T12:29:41.771 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.768+0000 7f6aa14520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:41.796 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.793+0000 7f6aa14520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:41.798 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.794+0000 7f6aa14520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.795+0000 7f6aa14520c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.795+0000 7f6aa14520c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:41.799 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.795+0000 7f6aa14520c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:29:41.863 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:41.859+0000 7f6aa14520c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:29:46.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:29:46.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:29:46.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:29:46.516 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:29:46.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:29:46.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:29:46.989 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:29:46.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:29:46.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:29:46.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:29:46.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:29:47.007 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:47.003+0000 7f471973a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:47.009 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:47.005+0000 7f471973a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:47.011 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:47.007+0000 7f471973a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:29:47.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:29:47.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:29:47.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:29:47.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:29:47.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:29:47.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:47.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:29:47.349 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:29:47.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:47.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:47.525 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:47.521+0000 7f471973a0c0 -1 Falling back to public interface 2021-05-12T12:29:47.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:48.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:48.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:48.727 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:29:48.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:29:48.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:48.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:49.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:29:49.793 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:29:49.789+0000 7f471973a0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:29:50.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:29:50.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:29:50.093 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:29:50.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:29:50.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:29:50.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:29:50.470 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178294,v1:127.0.0.1:6819/178294] [v2:127.0.0.1:6820/178294,v1:127.0.0.1:6821/178294] exists,up 67a0c8dd-c36d-480c-a625-fc4d701f2621 2021-05-12T12:29:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:29:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:29:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:29:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2021-05-12T12:29:50.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T12:29:50.908 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' does not exist 2021-05-12T12:29:50.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T12:29:50.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T12:29:51.670 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'rbd' already exists 2021-05-12T12:29:51.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:29:52.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T12:29:54.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2021-05-12T12:29:54.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2021-05-12T12:29:54.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2021-05-12T12:29:54.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T12:29:54.219 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:29:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2021-05-12T12:29:54.345 INFO:tasks.workunit.client.0.smithi090.stdout:{} 2021-05-12T12:29:54.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2021-05-12T12:29:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T12:29:54.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2021-05-12T12:29:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T12:29:54.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T12:29:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:29:54.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:29:54.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:29:54.820 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:29:54.820 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:29:54.820 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:29:54.821 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:54.821 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:29:54.821 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:29:54.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:29:54.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:29:54.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:29:55.186 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:29:55.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:29:55.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:29:55.596 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:29:55.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:05.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:30:05.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:30:05.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:30:06.088 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:30:06.088 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:30:06.088 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:30:06.089 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:06.089 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:30:06.089 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:30:06.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:30:06.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:30:06.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:30:06.447 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:06.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:30:06.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:30:06.829 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:30:06.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:16.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:30:16.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:30:16.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:30:17.317 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:30:17.317 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:30:17.318 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:30:17.318 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:17.318 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:30:17.318 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:30:17.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:30:17.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:30:17.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:30:17.682 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:17.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:30:17.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:30:18.060 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:30:18.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:28.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T12:30:28.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T12:30:28.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T12:30:28.547 INFO:tasks.workunit.client.0.smithi090.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T12:30:28.547 INFO:tasks.workunit.client.0.smithi090.stdout:-1 0.29306 root default 2021-05-12T12:30:28.548 INFO:tasks.workunit.client.0.smithi090.stdout:-3 0.29306 host smithi090 2021-05-12T12:30:28.548 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:28.549 INFO:tasks.workunit.client.0.smithi090.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T12:30:28.549 INFO:tasks.workunit.client.0.smithi090.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T12:30:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T12:30:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T12:30:28.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T12:30:28.933 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:30:28.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T12:30:28.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T12:30:29.374 INFO:tasks.workunit.client.0.smithi090.stderr:marked down osd.0. 2021-05-12T12:30:29.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T12:30:39.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2021-05-12T12:30:39.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2021-05-12T12:30:39.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2021-05-12T12:30:39.762 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T12:30:39.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:30:39.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:30:39.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:30:39.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:30:39.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:30:39.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:30:39.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:30:39.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:30:39.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:30:39.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:30:39.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:30:39.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:30:39.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:30:39.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:30:39.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:30:39.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:30:39.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:39.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:30:39.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:30:39.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:39.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:30:39.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:30:39.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:30:39.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:30:39.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:39.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:39.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:30:39.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:30:39.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:30:39.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T12:30:39.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T12:30:39.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T12:30:39.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T12:30:39.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:30:39.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:30:39.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:30:39.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:30:39.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:30:39.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:30:39.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:30:39.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:30:39.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:30:39.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:30:39.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:30:39.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:30:39.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:30:39.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:30:39.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:30:39.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:39.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:30:39.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:30:39.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:30:39.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:30:39.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:30:39.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:30:39.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:30:39.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:39.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:39.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:30:39.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:30:39.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:30:39.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T12:30:39.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:30:39.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:39.900 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:39.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164437 2021-05-12T12:30:39.900 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:30:39.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:30:39.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:30:39.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T12:30:39.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2021-05-12T12:30:39.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2021-05-12T12:30:39.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2021-05-12T12:30:39.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T12:30:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:30:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:30:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:30:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T12:30:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T12:30:40.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:30:40.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:40.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:40.260 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:40.260 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:40.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:30:40.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:30:40.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:30:40.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:30:40.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:30:40.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:30:40.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:30:40.517 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:30:40.518 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:30:40.518 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:30:40.518 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:30:40.519 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:40.519 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:40.520 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:30:40.520 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:30:40.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get fsid 2021-05-12T12:30:40.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:30:40.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:30:40.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:30:40.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:30:40.609 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:30:40.609 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:30:40.610 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:30:40.610 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:30:40.610 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:30:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164437/ceph-mon.a.asok 2021-05-12T12:30:40.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:30:40.612 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164437/ceph-mon.a.asok config get mon_host 2021-05-12T12:30:40.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2021-05-12T12:30:40.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T12:30:40.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:30:40.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:30:40.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:30:40.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T12:30:40.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:30:41.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:30:41.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:41.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:41.594 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:41.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:30:41.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:30:41.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2021-05-12T12:30:41.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:41.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:41.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:30:41.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:41.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T12:30:41.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:41.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:41.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:41.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:41.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T12:30:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T12:30:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:41.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:41.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:41.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:41.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:41.628 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:41.629 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:41.629 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:41.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:41.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:41.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:41.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:41.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:41.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:41.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:41.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:41.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:41.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:41.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:41.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:41.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T12:30:41.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:41.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=399be465-2679-4e47-8863-d67d1d8e3928 2021-05-12T12:30:41.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 399be465-2679-4e47-8863-d67d1d8e3928' 2021-05-12T12:30:41.637 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 399be465-2679-4e47-8863-d67d1d8e3928 2021-05-12T12:30:41.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:41.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBxyptgzRNzJRAAOJ6c2QmPVsGuPr2OaksFWw== 2021-05-12T12:30:41.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBxyptgzRNzJRAAOJ6c2QmPVsGuPr2OaksFWw=="}' 2021-05-12T12:30:41.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 399be465-2679-4e47-8863-d67d1d8e3928 -i td/osd-markdown/0/new.json 2021-05-12T12:30:41.936 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:30:41.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T12:30:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBxyptgzRNzJRAAOJ6c2QmPVsGuPr2OaksFWw== --osd-uuid 399be465-2679-4e47-8863-d67d1d8e3928 2021-05-12T12:30:41.968 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.964+0000 7f9b9fa210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:41.970 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.966+0000 7f9b9fa210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:41.972 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.968+0000 7f9b9fa210c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:41.973 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.969+0000 7f9b9fa210c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:41.973 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.969+0000 7f9b9fa210c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:41.973 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:41.969+0000 7f9b9fa210c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T12:30:42.011 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:42.007+0000 7f9b9fa210c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T12:30:46.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T12:30:46.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:30:46.625 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:30:46.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:30:46.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:30:47.092 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:30:47.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:30:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:30:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:30:47.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:30:47.111 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:47.107+0000 7f82f04c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:47.113 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:47.109+0000 7f82f04c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:47.114 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:47.111+0000 7f82f04c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:47.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:30:47.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:30:47.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:30:47.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:30:47.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:30:47.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:47.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:30:47.448 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:30:47.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:47.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:47.630 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:47.625+0000 7f82f04c70c0 -1 Falling back to public interface 2021-05-12T12:30:47.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:48.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:48.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:48.804 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:30:48.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:30:48.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:48.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:49.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:49.794 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:49.790+0000 7f82f04c70c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:30:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:30:50.166 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:30:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:50.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:30:50.534 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180542,v1:127.0.0.1:6803/180542] [v2:127.0.0.1:6804/180542,v1:127.0.0.1:6805/180542] exists,up 399be465-2679-4e47-8863-d67d1d8e3928 2021-05-12T12:30:50.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:30:50.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:30:50.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:30:50.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2021-05-12T12:30:50.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:50.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:50.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:30:50.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:50.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T12:30:50.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:50.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:50.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:50.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:50.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T12:30:50.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T12:30:50.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:50.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:50.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:50.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:50.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:50.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:50.544 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:50.544 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:50.545 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:50.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:50.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:50.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:50.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:50.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:50.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:50.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:50.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:50.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T12:30:50.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:50.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=34d3ada8-2b47-4b24-8c5f-e17e4305d3f6 2021-05-12T12:30:50.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 34d3ada8-2b47-4b24-8c5f-e17e4305d3f6' 2021-05-12T12:30:50.549 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 34d3ada8-2b47-4b24-8c5f-e17e4305d3f6 2021-05-12T12:30:50.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:50.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB6yptgbCStIBAA07pa9MCCS100U8IIrm/FeA== 2021-05-12T12:30:50.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB6yptgbCStIBAA07pa9MCCS100U8IIrm/FeA=="}' 2021-05-12T12:30:50.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 34d3ada8-2b47-4b24-8c5f-e17e4305d3f6 -i td/osd-markdown/1/new.json 2021-05-12T12:30:51.061 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:30:51.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T12:30:51.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB6yptgbCStIBAA07pa9MCCS100U8IIrm/FeA== --osd-uuid 34d3ada8-2b47-4b24-8c5f-e17e4305d3f6 2021-05-12T12:30:51.093 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.089+0000 7f466e7900c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:51.147 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.143+0000 7f466e7900c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:51.149 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.144+0000 7f466e7900c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:51.150 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.146+0000 7f466e7900c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:51.150 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.146+0000 7f466e7900c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:51.151 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.146+0000 7f466e7900c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T12:30:51.240 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:51.237+0000 7f466e7900c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T12:30:55.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T12:30:55.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:30:55.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:30:55.809 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:30:55.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:30:56.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:30:56.287 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:30:56.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:56.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:30:56.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:30:56.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:30:56.305 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:56.302+0000 7f67c33b30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:56.307 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:56.304+0000 7f67c33b30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:56.308 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:56.306+0000 7f67c33b30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:30:56.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:30:56.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:30:56.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:30:56.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:30:56.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:30:56.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:56.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:30:56.672 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:30:56.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:56.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:56.825 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:56.821+0000 7f67c33b30c0 -1 Falling back to public interface 2021-05-12T12:30:57.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:58.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:58.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:58.023 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:30:58.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:30:58.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:58.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:58.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:30:59.035 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:30:59.032+0000 7f67c33b30c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:30:59.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:30:59.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:30:59.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:30:59.390 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:30:59.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:30:59.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:30:59.752 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/181935,v1:127.0.0.1:6811/181935] [v2:127.0.0.1:6812/181935,v1:127.0.0.1:6813/181935] exists,up 34d3ada8-2b47-4b24-8c5f-e17e4305d3f6 2021-05-12T12:30:59.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:30:59.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:30:59.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:30:59.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2021-05-12T12:30:59.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T12:30:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:30:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:30:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:30:59.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T12:30:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T12:30:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:30:59.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:30:59.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:30:59.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T12:30:59.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T12:30:59.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:30:59.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:30:59.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T12:30:59.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:30:59.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:30:59.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:30:59.762 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:30:59.762 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:30:59.762 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:30:59.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:59.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' 2021-05-12T12:30:59.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:30:59.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:30:59.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:30:59.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T12:30:59.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T12:30:59.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:30:59.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:30:59.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:30:59.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:30:59.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:30:59.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T12:30:59.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:30:59.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e1fa372f-b6ce-475a-beeb-3d59cb6603c6 2021-05-12T12:30:59.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 e1fa372f-b6ce-475a-beeb-3d59cb6603c6' 2021-05-12T12:30:59.766 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 e1fa372f-b6ce-475a-beeb-3d59cb6603c6 2021-05-12T12:30:59.767 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:30:59.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCDyptgt/GdLRAAsys3wgs1xq7BMDq8lJaIBQ== 2021-05-12T12:30:59.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCDyptgt/GdLRAAsys3wgs1xq7BMDq8lJaIBQ=="}' 2021-05-12T12:30:59.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e1fa372f-b6ce-475a-beeb-3d59cb6603c6 -i td/osd-markdown/2/new.json 2021-05-12T12:31:00.325 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:31:00.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T12:31:00.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCDyptgt/GdLRAAsys3wgs1xq7BMDq8lJaIBQ== --osd-uuid e1fa372f-b6ce-475a-beeb-3d59cb6603c6 2021-05-12T12:31:00.359 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.356+0000 7fb86c40c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:00.401 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.398+0000 7fb86c40c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:00.403 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.400+0000 7fb86c40c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:00.404 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.401+0000 7fb86c40c0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:31:00.405 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.401+0000 7fb86c40c0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:31:00.405 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.401+0000 7fb86c40c0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T12:31:00.469 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:00.465+0000 7fb86c40c0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T12:31:05.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T12:31:05.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:05.558 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:31:05.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:31:05.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:06.036 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:31:06.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:31:06.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=08a61bda-b763-487b-9899-c1f477ecb9ae --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164437/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:06.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:06.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:06.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:06.056 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:06.052+0000 7fa7063bd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:06.057 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:06.054+0000 7fa7063bd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:06.059 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:06.056+0000 7fa7063bd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:06.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:31:06.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:06.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:31:06.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:06.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:06.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:06.413 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:06.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:06.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:06.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:31:06.575 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:06.572+0000 7fa7063bd0c0 -1 Falling back to public interface 2021-05-12T12:31:06.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:07.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:07.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:07.770 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:31:07.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:07.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:07.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:31:08.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:08.789 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:08.786+0000 7fa7063bd0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:31:09.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:09.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:09.138 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:31:09.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:09.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:09.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:31:09.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:10.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:10.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:10.504 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:31:10.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:31:10.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:10.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:31:10.879 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/183320,v1:127.0.0.1:6819/183320] [v2:127.0.0.1:6820/183320,v1:127.0.0.1:6821/183320] exists,up e1fa372f-b6ce-475a-beeb-3d59cb6603c6 2021-05-12T12:31:10.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:10.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:10.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:10.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2021-05-12T12:31:10.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=180542 2021-05-12T12:31:10.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 180542 2021-05-12T12:31:10.886 INFO:tasks.workunit.client.0.smithi090.stdout: PID TTY TIME CMD 2021-05-12T12:31:10.886 INFO:tasks.workunit.client.0.smithi090.stdout: 180542 ? 00:00:00 ceph-osd 2021-05-12T12:31:10.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2021-05-12T12:31:10.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2021-05-12T12:31:10.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2021-05-12T12:31:11.238 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T12:31:11.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2021-05-12T12:31:11.741 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already stopped. 2021-05-12T12:31:11.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2021-05-12T12:31:26.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2021-05-12T12:31:26.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2021-05-12T12:31:26.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2021-05-12T12:31:27.115 INFO:tasks.workunit.client.0.smithi090.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T12:31:27.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 180542 2021-05-12T12:31:27.120 INFO:tasks.workunit.client.0.smithi090.stdout: PID TTY TIME CMD 2021-05-12T12:31:27.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T12:31:27.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:31:27.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:27.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:31:27.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:27.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:27.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:27.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:27.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:27.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:27.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:27.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:27.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:27.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:27.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:27.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:27.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:27.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:27.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:27.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:27.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:31:27.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:27.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:31:27.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:31:27.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:27.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:27.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:31:27.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-markdown 0 2021-05-12T12:31:27.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T12:31:27.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:31:27.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T12:31:27.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:27.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:27.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:27.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:27.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:27.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:27.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:27.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:27.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:27.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:27.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:27.247 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:27.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:27.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:27.247 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:27.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:31:27.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T12:31:27.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:27.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164437 2021-05-12T12:31:27.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164437 2021-05-12T12:31:27.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:27.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:27.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:31:27.253 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:31:27.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:31:27.287 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2021-05-12T12:31:27.288 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2021-05-12T12:31:27.349 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:31:27.356 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:31:27.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:27.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:27.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:31:27.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:31:27.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:31:27.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:31:27.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:31:27.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:31:27.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-reactivate 2021-05-12T12:31:27.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2021-05-12T12:31:27.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2021-05-12T12:31:27.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2021-05-12T12:31:27.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2021-05-12T12:31:27.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2021-05-12T12:31:27.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2021-05-12T12:31:27.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none ' 2021-05-12T12:31:27.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:27.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2021-05-12T12:31:27.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:31:27.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2021-05-12T12:31:27.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2021-05-12T12:31:27.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2021-05-12T12:31:27.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2021-05-12T12:31:27.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2021-05-12T12:31:27.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:27.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:27.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:27.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:27.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:27.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:27.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:27.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:27.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:27.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:27.371 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:27.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:27.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:27.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:27.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:27.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:27.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:27.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:27.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.376 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:27.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184051 2021-05-12T12:31:27.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:27.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:27.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2021-05-12T12:31:27.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:31:27.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:27.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184051 2021-05-12T12:31:27.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:31:27.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:31:27.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:31:27.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:31:27.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2021-05-12T12:31:27.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2021-05-12T12:31:27.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2021-05-12T12:31:27.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2021-05-12T12:31:27.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2021-05-12T12:31:27.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:31:27.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:31:27.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:31:27.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2021-05-12T12:31:27.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2021-05-12T12:31:27.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:31:27.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:27.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:27.735 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:27.735 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.736 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:27.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:27.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:31:27.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:31:27.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:31:27.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:27.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:27.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:31:27.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:31:27.989 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:27.989 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:27.989 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:27.990 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:27.990 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:27.990 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:27.990 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184051/ceph-mon.a.asok 2021-05-12T12:31:27.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:27.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.184051/ceph-mon.a.asok config get fsid 2021-05-12T12:31:28.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:31:28.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:28.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:28.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:31:28.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:31:28.083 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:28.083 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:28.083 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:28.084 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:28.084 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:28.084 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:28.084 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184051/ceph-mon.a.asok 2021-05-12T12:31:28.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:28.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.184051/ceph-mon.a.asok config get mon_host 2021-05-12T12:31:28.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2021-05-12T12:31:28.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2021-05-12T12:31:28.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:31:28.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:31:28.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:31:28.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2021-05-12T12:31:28.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:31:29.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:31:29.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:29.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:29.049 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:29.049 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:29.049 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:29.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:29.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:29.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:29.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2021-05-12T12:31:29.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2021-05-12T12:31:29.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:29.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:31:29.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:29.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2021-05-12T12:31:29.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:29.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:29.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:29.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T12:31:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T12:31:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T12:31:29.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:29.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:29.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:29.077 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:29.078 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:29.078 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:29.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:29.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:29.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:29.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:29.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:29.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T12:31:29.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T12:31:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:29.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:29.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-reactivate/0 2021-05-12T12:31:29.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:29.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1cc9cb0f-510e-425f-8aac-208c188bfc4a 2021-05-12T12:31:29.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 1cc9cb0f-510e-425f-8aac-208c188bfc4a' 2021-05-12T12:31:29.082 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 1cc9cb0f-510e-425f-8aac-208c188bfc4a 2021-05-12T12:31:29.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQChyptgUe7hBBAAL4+zHxfzweTf0eLpJPB8vg== 2021-05-12T12:31:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQChyptgUe7hBBAAL4+zHxfzweTf0eLpJPB8vg=="}' 2021-05-12T12:31:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1cc9cb0f-510e-425f-8aac-208c188bfc4a -i td/osd-reactivate/0/new.json 2021-05-12T12:31:29.394 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:29.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-reactivate/0/new.json 2021-05-12T12:31:29.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQChyptgUe7hBBAAL4+zHxfzweTf0eLpJPB8vg== --osd-uuid 1cc9cb0f-510e-425f-8aac-208c188bfc4a 2021-05-12T12:31:29.426 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.423+0000 7ff46b34d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:29.427 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.425+0000 7ff46b34d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:29.429 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.426+0000 7ff46b34d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:29.430 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.427+0000 7ff46b34d0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:29.430 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.427+0000 7ff46b34d0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:29.431 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.427+0000 7ff46b34d0c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T12:31:29.469 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:29.466+0000 7ff46b34d0c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2021-05-12T12:31:34.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-reactivate/0/keyring 2021-05-12T12:31:34.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:34.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:31:34.080 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:31:34.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:34.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:31:34.547 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:31:34.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:34.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:34.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:34.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:34.566 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:34.563+0000 7f61db3490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:34.567 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:34.564+0000 7f61db3490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:34.568 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:34.566+0000 7f61db3490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:34.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:31:34.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:34.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:34.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:34.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:34.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:34.907 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:34.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:34.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:34.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:35.083 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:35.079+0000 7f61db3490c0 -1 Falling back to public interface 2021-05-12T12:31:35.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:36.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:36.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:36.282 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:31:36.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:36.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:36.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:36.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:37.241 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:37.237+0000 7f61db3490c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:31:37.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:37.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:37.643 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:31:37.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:37.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:37.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:38.010 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185171,v1:127.0.0.1:6803/185171] [v2:127.0.0.1:6804/185171,v1:127.0.0.1:6805/185171] exists,up 1cc9cb0f-510e-425f-8aac-208c188bfc4a 2021-05-12T12:31:38.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:38.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:38.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:38.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2021-05-12T12:31:38.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:38.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:38.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:38.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:38.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:38.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:38.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2021-05-12T12:31:38.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2021-05-12T12:31:38.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2021-05-12T12:31:38.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2021-05-12T12:31:38.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-reactivate 2021-05-12T12:31:38.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T12:31:38.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T12:31:38.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T12:31:38.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-reactivate/0 2021-05-12T12:31:38.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T12:31:38.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:38.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:38.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:38.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T12:31:38.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T12:31:38.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:38.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T12:31:38.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T12:31:38.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T12:31:38.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:38.135 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:38.135 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:38.136 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:38.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:38.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' 2021-05-12T12:31:38.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:38.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T12:31:38.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T12:31:38.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:38.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:38.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:38.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T12:31:38.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T12:31:38.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-reactivate/0 2021-05-12T12:31:38.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T12:31:38.140 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:31:38.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=a910afd7-f0a0-4ce3-b826-b4fa36c6dc75 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184051/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:38.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-reactivate/0/whoami 2021-05-12T12:31:38.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T12:31:38.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T12:31:38.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T12:31:38.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T12:31:38.143 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:38.131+0000 7f3bfe1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:38.143 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:38.133+0000 7f3bfe1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:38.144 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:38.135+0000 7f3bfe1d80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:38.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T12:31:38.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:38.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:38.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:38.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:38.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:38.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:38.482 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:38.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:38.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:38.675 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:38.671+0000 7f3bfe1d80c0 -1 Falling back to public interface 2021-05-12T12:31:38.825 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185171,v1:127.0.0.1:6803/185171] [v2:127.0.0.1:6804/185171,v1:127.0.0.1:6805/185171] exists,up 1cc9cb0f-510e-425f-8aac-208c188bfc4a 2021-05-12T12:31:38.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:38.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:38.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:38.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2021-05-12T12:31:38.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:38.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:38.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:38.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:38.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:38.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:38.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:38.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:38.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:38.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:38.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:38.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:38.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:38.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:38.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:38.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:38.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:38.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:38.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:38.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:38.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:38.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:38.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:38.938 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:38.938 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:38.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184051 2021-05-12T12:31:38.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:38.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:38.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T12:31:38.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-reactivate 0 2021-05-12T12:31:38.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T12:31:38.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T12:31:38.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T12:31:38.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:38.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:38.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:38.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:38.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:38.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:38.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:38.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:38.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:38.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:38.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:38.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:38.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:38.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:38.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:38.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:38.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:38.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T12:31:38.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T12:31:38.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:38.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:38.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184051 2021-05-12T12:31:38.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184051 2021-05-12T12:31:38.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:38.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:38.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T12:31:38.952 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T12:31:38.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T12:31:39.020 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2021-05-12T12:31:39.020 DEBUG:teuthology.orchestra.run.smithi090:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2021-05-12T12:31:39.082 INFO:tasks.workunit.client.0.smithi090.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T12:31:39.109 INFO:tasks.workunit.client.0.smithi090.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T12:31:39.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:39.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T12:31:39.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T12:31:39.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T12:31:39.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T12:31:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T12:31:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T12:31:39.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T12:31:39.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-recovery-prio 2021-05-12T12:31:39.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2021-05-12T12:31:39.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-05-12T12:31:39.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-05-12T12:31:39.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2021-05-12T12:31:39.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2021-05-12T12:31:39.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none ' 2021-05-12T12:31:39.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:39.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2021-05-12T12:31:39.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2021-05-12T12:31:39.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2021-05-12T12:31:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2021-05-12T12:31:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2021-05-12T12:31:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2021-05-12T12:31:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2021-05-12T12:31:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2021-05-12T12:31:39.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2021-05-12T12:31:39.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T12:31:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2021-05-12T12:31:39.121 INFO:tasks.workunit.client.0.smithi090.stderr:TEST_recovery_priority' 2021-05-12T12:31:39.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-05-12T12:31:39.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-05-12T12:31:39.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-05-12T12:31:39.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:31:39.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:31:39.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:31:39.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:31:39.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:31:39.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:31:39.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:31:39.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:31:39.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:31:39.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:31:39.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:31:39.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:31:39.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:31:39.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:31:39.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:39.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:31:39.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:31:39.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:31:39.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:31:39.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:31:39.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:31:39.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:31:39.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:39.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:39.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.185878 2021-05-12T12:31:39.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:31:39.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:31:39.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-05-12T12:31:39.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:31:39.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:39.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:39.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.185878 2021-05-12T12:31:39.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:31:39.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T12:31:39.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T12:31:39.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:31:39.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-05-12T12:31:39.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2021-05-12T12:31:39.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2021-05-12T12:31:39.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2021-05-12T12:31:39.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2021-05-12T12:31:39.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:31:39.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:31:39.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:31:39.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-05-12T12:31:39.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-05-12T12:31:39.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:31:39.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:39.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:39.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:39.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:31:39.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:31:39.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:31:39.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:39.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:39.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:31:39.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:31:39.756 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:39.756 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:39.756 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:39.756 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:39.757 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:39.757 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:39.757 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-mon.a.asok 2021-05-12T12:31:39.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:39.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185878/ceph-mon.a.asok config get fsid 2021-05-12T12:31:39.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:31:39.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:31:39.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:31:39.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:31:39.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:31:39.852 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:31:39.852 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:31:39.852 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:31:39.853 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:31:39.853 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:39.854 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:39.854 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-mon.a.asok 2021-05-12T12:31:39.854 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:31:39.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185878/ceph-mon.a.asok config get mon_host 2021-05-12T12:31:39.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2021-05-12T12:31:39.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-05-12T12:31:39.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:31:39.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:31:39.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:31:39.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-05-12T12:31:39.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:31:40.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:31:40.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:40.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:40.949 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:40.949 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:40.950 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:40.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:40.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:40.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:31:40.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2021-05-12T12:31:40.971 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2021-05-12T12:31:40.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2021-05-12T12:31:40.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:31:40.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2021-05-12T12:31:40.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:31:40.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:40.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:31:40.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:40.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-05-12T12:31:40.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:40.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:40.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:40.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:40.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-05-12T12:31:40.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-05-12T12:31:40.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:40.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:40.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:31:40.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:40.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:40.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:40.991 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:40.991 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:40.992 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:40.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:40.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:40.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:40.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:40.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:40.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:31:40.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:31:40.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:41.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:41.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:41.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:41.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:41.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/0 2021-05-12T12:31:41.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:41.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=cc48ce63-21d1-49b6-b850-5752f8037518 2021-05-12T12:31:41.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 cc48ce63-21d1-49b6-b850-5752f8037518' 2021-05-12T12:31:41.004 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 cc48ce63-21d1-49b6-b850-5752f8037518 2021-05-12T12:31:41.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:41.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCsyptgXoPcOhAAc1jYrWgDa5nIB1jo6gRC5w== 2021-05-12T12:31:41.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCsyptgXoPcOhAAc1jYrWgDa5nIB1jo6gRC5w=="}' 2021-05-12T12:31:41.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new cc48ce63-21d1-49b6-b850-5752f8037518 -i td/osd-recovery-prio/0/new.json 2021-05-12T12:31:41.301 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:41.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/0/new.json 2021-05-12T12:31:41.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCsyptgXoPcOhAAc1jYrWgDa5nIB1jo6gRC5w== --osd-uuid cc48ce63-21d1-49b6-b850-5752f8037518 2021-05-12T12:31:41.336 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.332+0000 7f4b7b5d50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:41.337 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.333+0000 7f4b7b5d50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:41.338 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.335+0000 7f4b7b5d50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:41.339 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.335+0000 7f4b7b5d50c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:41.340 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.335+0000 7f4b7b5d50c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:41.340 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.335+0000 7f4b7b5d50c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:31:41.386 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:41.382+0000 7f4b7b5d50c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-05-12T12:31:45.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-05-12T12:31:45.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:45.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:31:45.993 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:31:45.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:46.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:31:46.464 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:31:46.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:46.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:46.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:46.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:46.486 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:46.480+0000 7f86053070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:46.487 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:46.482+0000 7f86053070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:46.488 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:46.484+0000 7f86053070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:46.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:31:46.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:46.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:31:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:46.823 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:46.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:46.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:47.007 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:47.000+0000 7f86053070c0 -1 Falling back to public interface 2021-05-12T12:31:47.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:48.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:48.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:48.205 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:31:48.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:48.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:48.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:48.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:49.176 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:49.162+0000 7f86053070c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:31:49.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:49.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:49.566 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:31:49.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:49.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:49.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:31:49.978 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/187001,v1:127.0.0.1:6803/187001] [v2:127.0.0.1:6804/187001,v1:127.0.0.1:6805/187001] exists,up cc48ce63-21d1-49b6-b850-5752f8037518 2021-05-12T12:31:49.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:49.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:49.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:49.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:31:49.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2021-05-12T12:31:49.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:31:49.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:31:49.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:31:49.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:31:49.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-05-12T12:31:49.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:31:49.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:31:49.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:31:49.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:31:49.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-05-12T12:31:49.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-05-12T12:31:49.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:31:49.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:31:49.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:31:49.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:31:49.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:31:49.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:31:49.988 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:31:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:31:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:31:49.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:31:49.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:31:49.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:31:49.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:31:49.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:31:49.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:31:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:31:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:31:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:31:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:31:49.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/1 2021-05-12T12:31:49.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:31:49.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=20741e48-3f9a-4c0f-888a-87e9ef4febfc 2021-05-12T12:31:49.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 20741e48-3f9a-4c0f-888a-87e9ef4febfc' 2021-05-12T12:31:49.993 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 20741e48-3f9a-4c0f-888a-87e9ef4febfc 2021-05-12T12:31:49.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:31:49.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC1yptgPNAsOxAAfXQfS1QqV85QW1U8KJgCng== 2021-05-12T12:31:49.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC1yptgPNAsOxAAfXQfS1QqV85QW1U8KJgCng=="}' 2021-05-12T12:31:49.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 20741e48-3f9a-4c0f-888a-87e9ef4febfc -i td/osd-recovery-prio/1/new.json 2021-05-12T12:31:50.471 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:31:50.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/1/new.json 2021-05-12T12:31:50.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC1yptgPNAsOxAAfXQfS1QqV85QW1U8KJgCng== --osd-uuid 20741e48-3f9a-4c0f-888a-87e9ef4febfc 2021-05-12T12:31:50.509 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.502+0000 7f7c453c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:50.509 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.504+0000 7f7c453c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:50.510 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.505+0000 7f7c453c70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:50.510 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.506+0000 7f7c453c70c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:50.510 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.506+0000 7f7c453c70c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:50.511 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.506+0000 7f7c453c70c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:31:50.567 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:50.559+0000 7f7c453c70c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-05-12T12:31:55.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-05-12T12:31:55.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:31:55.216 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:31:55.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:31:55.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:31:55.695 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:31:55.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:31:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:31:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:31:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:31:55.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:31:55.716 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:55.713+0000 7f4fd19670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:55.717 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:55.715+0000 7f4fd19670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:55.722 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:55.716+0000 7f4fd19670c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:31:56.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:31:56.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:31:56.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:31:56.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:31:56.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:31:56.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:56.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:31:56.055 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:31:56.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:56.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:56.236 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:56.233+0000 7f4fd19670c0 -1 Falling back to public interface 2021-05-12T12:31:56.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:57.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:57.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:57.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:31:57.435 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:31:57.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:57.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:57.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:31:58.440 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:31:58.437+0000 7f4fd19670c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:31:58.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:31:58.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:31:58.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:31:58.806 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:31:58.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:31:58.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:31:59.170 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/188394,v1:127.0.0.1:6811/188394] [v2:127.0.0.1:6812/188394,v1:127.0.0.1:6813/188394] exists,up 20741e48-3f9a-4c0f-888a-87e9ef4febfc 2021-05-12T12:31:59.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:31:59.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:31:59.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:31:59.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:31:59.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:31:59.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2021-05-12T12:31:59.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-05-12T12:31:59.668 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test1' already exists 2021-05-12T12:31:59.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:32:00.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T12:32:01.584 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 2 2021-05-12T12:32:01.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:01.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2021-05-12T12:32:01.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-05-12T12:32:02.103 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test2' already exists 2021-05-12T12:32:02.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:32:03.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T12:32:03.793 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 2 size to 2 2021-05-12T12:32:03.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:03.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2021-05-12T12:32:03.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-05-12T12:32:04.348 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test3' already exists 2021-05-12T12:32:04.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:32:05.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2021-05-12T12:32:06.030 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 3 size to 2 2021-05-12T12:32:06.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2021-05-12T12:32:11.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:32:11.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:32:11.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:32:11.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:32:11.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:32:11.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:32:11.043 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:32:11.043 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:32:11.043 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:32:11.043 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:32:11.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:32:11.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:32:11.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:32:11.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:32:11.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:32:11.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:32:11.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:32:11.720 INFO:tasks.workunit.client.0.smithi090.stderr:1' 2021-05-12T12:32:11.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:32:11.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:11.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:32:11.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T12:32:11.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T12:32:11.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T12:32:11.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:11.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:32:11.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T12:32:11.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T12:32:11.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-05-12T12:32:11.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:11.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T12:32:11.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:11.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:32:11.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T12:32:11.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:11.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T12:32:11.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T12:32:11.980 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836487 2021-05-12T12:32:11.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:12.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T12:32:12.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:32:13.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:32:13.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:13.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836487 2021-05-12T12:32:13.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:13.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T12:32:13.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:13.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:32:13.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T12:32:13.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:13.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T12:32:13.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T12:32:13.729 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672965 2021-05-12T12:32:13.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:32:14.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672965 -lt 42949672965 2021-05-12T12:32:14.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:32:14.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:14.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:14.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:32:14.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:32:14.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:32:14.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:32:14.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:32:14.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:32:14.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:32:14.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:32:14.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=3 2021-05-12T12:32:14.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:32:14.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:14.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:15.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 = 3 2021-05-12T12:32:15.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:32:15.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:32:15.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:32:15.694 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:32:15.695 INFO:tasks.workunit.client.0.smithi090.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:32:06.216488+0000 0'0 26:16 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:32:04.118326+0000 0'0 2021-05-12T12:32:04.118326+0000 0 2021-05-12T12:32:15.695 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:32:01.779461+0000 0'0 25:26 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:31:59.477756+0000 0'0 2021-05-12T12:31:59.477756+0000 0 2021-05-12T12:32:15.695 INFO:tasks.workunit.client.0.smithi090.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:32:04.366045+0000 0'0 25:23 [0,1] 0 [0,1] 0 0'0 2021-05-12T12:32:01.862272+0000 0'0 2021-05-12T12:32:01.862272+0000 0 2021-05-12T12:32:15.695 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:32:15.696 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:32:15.696 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:32:15.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2021-05-12T12:32:15.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2021-05-12T12:32:15.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2021-05-12T12:32:15.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2021-05-12T12:32:15.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2021-05-12T12:32:15.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2021-05-12T12:32:15.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2021-05-12T12:32:15.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2021-05-12T12:32:15.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2021-05-12T12:32:15.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2021-05-12T12:32:15.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:32:15.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:15.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2021-05-12T12:32:15.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T12:32:16.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:32:16.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2021-05-12T12:32:16.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:32:16.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2021-05-12T12:32:16.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2021-05-12T12:32:16.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2021-05-12T12:32:16.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2021-05-12T12:32:16.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2021-05-12T12:32:16.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2021-05-12T12:32:16.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2021-05-12T12:32:16.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:16.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2021-05-12T12:32:16.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T12:32:16.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:32:16.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2021-05-12T12:32:16.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:32:16.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2021-05-12T12:32:16.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2021-05-12T12:32:16.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2021-05-12T12:32:16.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2021-05-12T12:32:16.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2021-05-12T12:32:16.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2021-05-12T12:32:16.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2021-05-12T12:32:16.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2021-05-12T12:32:16.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2021-05-12T12:32:16.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2021-05-12T12:32:16.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2021-05-12T12:32:16.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2021-05-12T12:32:16.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:16.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-05-12T12:32:16.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:16.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-05-12T12:32:16.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T12:32:16.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-05-12T12:32:16.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2021-05-12T12:32:16.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-05-12T12:32:16.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-05-12T12:32:16.888 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test3' does not exist 2021-05-12T12:32:16.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2021-05-12T12:32:16.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2021-05-12T12:32:16.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2021-05-12T12:32:16.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2021-05-12T12:32:16.906 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2021-05-12T12:32:16.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2021-05-12T12:32:16.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-05-12T12:32:17.579 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 1 2021-05-12T12:32:17.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-05-12T12:32:18.312 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 recovery_priority to 1 2021-05-12T12:32:18.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-05-12T12:32:19.031 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 2 size to 1 2021-05-12T12:32:19.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-05-12T12:32:19.807 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 2 recovery_priority to 2 2021-05-12T12:32:19.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:32:19.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:32:19.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:32:19.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:32:19.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:32:19.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:32:19.820 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:32:19.820 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:32:19.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:32:19.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:32:19.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:32:19.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:32:19.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:32:19.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:32:19.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:32:19.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:32:20.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:32:20.284 INFO:tasks.workunit.client.0.smithi090.stderr:1' 2021-05-12T12:32:20.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:32:20.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:20.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:32:20.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T12:32:20.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T12:32:20.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T12:32:20.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:32:20.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:32:20.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T12:32:20.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T12:32:20.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2021-05-12T12:32:20.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:20.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T12:32:20.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:20.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:32:20.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T12:32:20.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:20.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T12:32:20.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T12:32:20.556 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836490 2021-05-12T12:32:20.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:20.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T12:32:20.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:32:21.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:32:21.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:32:22.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836490 2021-05-12T12:32:22.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:32:22.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T12:32:22.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:32:22.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:32:22.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T12:32:22.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:32:22.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T12:32:22.266 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949672969 2021-05-12T12:32:22.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T12:32:22.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:32:22.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T12:32:22.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:32:22.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:22.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T12:32:23.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:32:23.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:32:23.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:32:23.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:32:23.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:32:23.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:32:23.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:32:23.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T12:32:23.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:32:23.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:32:23.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:32:23.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T12:32:23.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:32:23.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:32:23.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-05-12T12:32:23.965 INFO:tasks.workunit.client.0.smithi090.stderr:10+0 records in 2021-05-12T12:32:23.966 INFO:tasks.workunit.client.0.smithi090.stderr:10+0 records out 2021-05-12T12:32:23.966 INFO:tasks.workunit.client.0.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0457746 s, 229 MB/s 2021-05-12T12:32:23.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2021-05-12T12:32:23.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T12:32:23.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T12:32:23.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:23.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-05-12T12:32:24.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:24.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-05-12T12:32:25.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:25.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-05-12T12:32:26.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:26.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-05-12T12:32:26.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:26.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-05-12T12:32:26.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:26.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-05-12T12:32:27.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:27.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-05-12T12:32:27.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:27.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-05-12T12:32:27.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:27.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-05-12T12:32:28.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:28.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-05-12T12:32:29.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:29.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-05-12T12:32:29.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:29.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-05-12T12:32:30.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:30.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-05-12T12:32:30.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:30.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-05-12T12:32:30.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:30.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-05-12T12:32:31.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:31.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-05-12T12:32:31.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:31.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-05-12T12:32:32.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:32.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-05-12T12:32:32.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:32.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-05-12T12:32:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-05-12T12:32:33.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:34.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-05-12T12:32:34.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:34.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-05-12T12:32:35.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:35.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-05-12T12:32:35.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:35.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-05-12T12:32:36.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:36.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-05-12T12:32:36.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:36.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-05-12T12:32:36.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:36.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-05-12T12:32:37.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:37.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-05-12T12:32:37.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:37.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-05-12T12:32:38.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:38.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-05-12T12:32:38.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:38.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-05-12T12:32:39.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:39.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-05-12T12:32:39.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:39.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-05-12T12:32:39.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:39.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-05-12T12:32:40.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:40.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-05-12T12:32:40.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:40.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-05-12T12:32:41.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:41.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-05-12T12:32:42.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:42.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-05-12T12:32:42.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:42.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-05-12T12:32:42.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:42.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-05-12T12:32:43.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:43.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-05-12T12:32:43.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:43.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-05-12T12:32:43.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:43.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-05-12T12:32:45.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:45.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-05-12T12:32:46.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:46.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-05-12T12:32:47.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:47.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-05-12T12:32:47.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:47.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-05-12T12:32:48.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:48.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-05-12T12:32:48.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:48.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-05-12T12:32:48.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:48.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-05-12T12:32:49.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:49.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-05-12T12:32:51.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:51.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-05-12T12:32:52.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:52.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-05-12T12:32:52.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:52.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-05-12T12:32:52.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:52.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-05-12T12:32:53.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:53.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-05-12T12:32:54.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:54.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-05-12T12:32:54.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:54.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-05-12T12:32:55.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:55.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-05-12T12:32:55.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:55.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-05-12T12:32:56.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:56.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-05-12T12:32:56.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:56.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-05-12T12:32:57.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:57.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-05-12T12:32:58.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:58.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-05-12T12:32:58.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:58.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-05-12T12:32:58.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:58.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-05-12T12:32:59.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:59.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-05-12T12:32:59.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:59.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-05-12T12:32:59.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:32:59.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-05-12T12:33:00.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:00.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-05-12T12:33:01.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:01.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-05-12T12:33:01.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:01.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-05-12T12:33:01.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:01.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-05-12T12:33:02.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:02.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-05-12T12:33:03.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:03.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-05-12T12:33:04.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:04.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-05-12T12:33:04.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:04.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-05-12T12:33:04.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:04.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-05-12T12:33:05.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:05.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-05-12T12:33:05.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:05.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-05-12T12:33:06.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:06.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-05-12T12:33:07.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:07.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-05-12T12:33:08.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:08.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-05-12T12:33:09.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:09.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-05-12T12:33:09.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:09.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-05-12T12:33:10.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:10.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-05-12T12:33:10.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:10.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-05-12T12:33:11.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:11.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-05-12T12:33:11.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:11.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-05-12T12:33:12.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:12.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-05-12T12:33:14.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:14.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-05-12T12:33:15.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:15.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-05-12T12:33:15.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:15.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-05-12T12:33:16.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:16.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-05-12T12:33:17.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:17.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-05-12T12:33:17.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:17.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-05-12T12:33:17.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:17.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-05-12T12:33:18.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:18.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-05-12T12:33:19.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:19.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-05-12T12:33:19.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:19.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-05-12T12:33:20.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:20.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-05-12T12:33:20.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:20.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-05-12T12:33:21.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:21.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-05-12T12:33:22.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:22.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-05-12T12:33:22.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:22.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-05-12T12:33:22.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:22.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-05-12T12:33:23.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:23.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-05-12T12:33:24.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:24.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-05-12T12:33:24.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:24.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-05-12T12:33:25.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:25.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-05-12T12:33:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:26.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-05-12T12:33:27.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:27.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-05-12T12:33:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:27.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-05-12T12:33:27.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:27.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-05-12T12:33:28.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:28.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-05-12T12:33:29.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:29.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-05-12T12:33:30.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:30.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-05-12T12:33:32.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:32.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-05-12T12:33:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-05-12T12:33:33.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:33.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-05-12T12:33:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-05-12T12:33:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:34.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-05-12T12:33:34.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:34.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-05-12T12:33:35.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:35.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-05-12T12:33:36.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:36.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-05-12T12:33:37.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:37.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-05-12T12:33:38.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:38.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-05-12T12:33:38.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:38.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-05-12T12:33:38.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:38.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-05-12T12:33:39.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:39.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-05-12T12:33:39.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:39.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-05-12T12:33:39.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:39.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-05-12T12:33:40.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:40.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-05-12T12:33:41.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:41.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-05-12T12:33:41.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:41.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-05-12T12:33:41.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:41.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-05-12T12:33:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:42.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-05-12T12:33:43.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:43.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-05-12T12:33:45.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:45.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-05-12T12:33:45.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:45.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-05-12T12:33:46.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:46.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-05-12T12:33:46.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:46.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-05-12T12:33:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-05-12T12:33:47.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:47.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-05-12T12:33:48.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-05-12T12:33:48.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-05-12T12:33:48.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:48.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-05-12T12:33:49.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:49.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-05-12T12:33:50.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:50.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-05-12T12:33:50.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:50.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-05-12T12:33:51.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:51.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-05-12T12:33:51.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:51.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-05-12T12:33:52.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:52.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-05-12T12:33:52.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:52.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-05-12T12:33:53.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:53.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-05-12T12:33:53.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:53.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-05-12T12:33:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-05-12T12:33:54.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-05-12T12:33:54.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:54.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-05-12T12:33:55.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:55.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-05-12T12:33:56.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:56.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-05-12T12:33:57.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:57.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-05-12T12:33:57.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:57.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-05-12T12:33:58.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-05-12T12:33:58.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:58.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-05-12T12:33:59.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:59.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-05-12T12:33:59.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:33:59.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-05-12T12:34:00.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-05-12T12:34:00.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-05-12T12:34:00.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-05-12T12:34:00.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:00.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-05-12T12:34:01.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:01.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-05-12T12:34:01.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T12:34:01.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2021-05-12T12:34:01.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T12:34:01.569 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T12:34:01.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:01.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2021-05-12T12:34:02.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:02.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2021-05-12T12:34:03.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:03.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2021-05-12T12:34:04.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:04.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2021-05-12T12:34:04.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:04.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2021-05-12T12:34:05.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:05.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2021-05-12T12:34:05.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:05.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2021-05-12T12:34:06.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:06.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2021-05-12T12:34:06.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:06.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2021-05-12T12:34:06.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:06.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2021-05-12T12:34:07.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:07.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2021-05-12T12:34:07.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:07.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2021-05-12T12:34:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2021-05-12T12:34:09.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:09.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2021-05-12T12:34:09.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:09.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2021-05-12T12:34:09.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:09.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2021-05-12T12:34:10.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:10.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2021-05-12T12:34:11.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:11.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2021-05-12T12:34:11.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:11.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2021-05-12T12:34:12.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:12.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2021-05-12T12:34:13.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:13.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2021-05-12T12:34:14.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:14.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2021-05-12T12:34:15.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:15.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2021-05-12T12:34:15.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:15.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2021-05-12T12:34:15.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:15.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2021-05-12T12:34:16.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:16.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2021-05-12T12:34:16.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:16.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2021-05-12T12:34:17.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:17.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2021-05-12T12:34:18.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:18.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2021-05-12T12:34:18.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:18.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2021-05-12T12:34:18.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:18.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2021-05-12T12:34:19.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:19.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2021-05-12T12:34:20.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:20.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2021-05-12T12:34:21.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:21.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2021-05-12T12:34:21.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:21.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2021-05-12T12:34:21.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:21.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2021-05-12T12:34:21.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:21.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2021-05-12T12:34:22.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:22.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2021-05-12T12:34:23.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:23.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2021-05-12T12:34:24.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:24.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2021-05-12T12:34:24.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:24.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2021-05-12T12:34:24.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:24.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2021-05-12T12:34:25.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:25.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2021-05-12T12:34:26.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:26.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2021-05-12T12:34:27.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:27.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2021-05-12T12:34:27.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:27.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2021-05-12T12:34:27.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:27.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2021-05-12T12:34:28.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:28.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2021-05-12T12:34:29.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:29.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2021-05-12T12:34:30.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:30.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2021-05-12T12:34:30.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:30.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2021-05-12T12:34:30.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:30.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2021-05-12T12:34:31.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:31.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2021-05-12T12:34:31.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:31.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2021-05-12T12:34:31.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:31.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2021-05-12T12:34:33.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:33.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2021-05-12T12:34:33.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:33.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2021-05-12T12:34:34.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:34.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2021-05-12T12:34:35.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:35.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2021-05-12T12:34:35.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:35.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2021-05-12T12:34:36.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:36.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2021-05-12T12:34:36.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:36.836 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2021-05-12T12:34:37.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:37.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2021-05-12T12:34:37.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:37.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2021-05-12T12:34:37.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:37.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2021-05-12T12:34:38.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:38.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2021-05-12T12:34:38.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:38.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2021-05-12T12:34:39.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:39.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2021-05-12T12:34:39.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:39.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2021-05-12T12:34:40.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:40.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2021-05-12T12:34:40.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:40.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2021-05-12T12:34:40.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:40.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2021-05-12T12:34:41.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:41.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2021-05-12T12:34:41.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:41.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2021-05-12T12:34:41.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:41.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2021-05-12T12:34:42.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:42.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2021-05-12T12:34:43.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:43.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2021-05-12T12:34:43.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:43.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2021-05-12T12:34:44.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:44.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2021-05-12T12:34:45.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:45.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2021-05-12T12:34:46.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:46.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2021-05-12T12:34:47.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:47.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2021-05-12T12:34:47.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:47.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2021-05-12T12:34:48.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:48.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2021-05-12T12:34:48.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:48.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2021-05-12T12:34:49.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:49.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2021-05-12T12:34:50.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:50.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2021-05-12T12:34:50.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:50.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2021-05-12T12:34:51.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:51.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2021-05-12T12:34:51.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:51.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2021-05-12T12:34:52.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:52.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2021-05-12T12:34:53.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:53.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2021-05-12T12:34:53.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:53.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2021-05-12T12:34:53.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:53.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2021-05-12T12:34:54.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:54.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2021-05-12T12:34:55.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:55.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2021-05-12T12:34:56.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:56.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2021-05-12T12:34:57.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:57.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2021-05-12T12:34:57.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:57.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2021-05-12T12:34:58.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:58.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2021-05-12T12:34:59.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:59.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2021-05-12T12:34:59.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:59.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2021-05-12T12:34:59.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:34:59.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2021-05-12T12:35:01.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:01.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2021-05-12T12:35:02.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:02.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2021-05-12T12:35:04.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:04.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2021-05-12T12:35:05.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:05.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2021-05-12T12:35:05.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:05.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2021-05-12T12:35:06.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:06.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2021-05-12T12:35:07.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:07.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2021-05-12T12:35:08.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:08.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2021-05-12T12:35:08.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:08.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2021-05-12T12:35:09.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:09.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2021-05-12T12:35:09.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:09.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2021-05-12T12:35:10.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:10.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2021-05-12T12:35:10.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:10.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2021-05-12T12:35:10.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:10.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2021-05-12T12:35:11.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:11.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2021-05-12T12:35:11.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:11.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2021-05-12T12:35:11.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:11.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2021-05-12T12:35:12.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:12.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2021-05-12T12:35:13.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:13.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2021-05-12T12:35:13.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:13.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2021-05-12T12:35:14.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:14.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2021-05-12T12:35:15.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:15.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2021-05-12T12:35:15.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:15.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2021-05-12T12:35:16.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:16.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2021-05-12T12:35:17.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:17.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2021-05-12T12:35:17.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:17.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2021-05-12T12:35:19.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:19.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2021-05-12T12:35:19.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:19.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2021-05-12T12:35:20.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:20.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2021-05-12T12:35:20.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:20.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2021-05-12T12:35:20.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:20.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2021-05-12T12:35:21.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:21.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2021-05-12T12:35:21.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:21.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2021-05-12T12:35:22.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:22.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2021-05-12T12:35:22.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:22.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2021-05-12T12:35:23.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:23.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2021-05-12T12:35:23.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:23.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2021-05-12T12:35:23.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:23.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2021-05-12T12:35:23.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:23.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2021-05-12T12:35:24.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:24.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2021-05-12T12:35:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:24.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2021-05-12T12:35:25.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:25.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2021-05-12T12:35:25.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:25.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2021-05-12T12:35:26.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:26.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2021-05-12T12:35:26.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:26.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2021-05-12T12:35:26.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:26.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2021-05-12T12:35:27.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:27.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2021-05-12T12:35:27.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:27.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2021-05-12T12:35:28.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:28.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2021-05-12T12:35:29.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:29.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2021-05-12T12:35:30.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:30.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2021-05-12T12:35:30.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:30.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2021-05-12T12:35:31.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:31.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2021-05-12T12:35:31.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:31.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2021-05-12T12:35:32.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:32.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2021-05-12T12:35:32.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:32.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2021-05-12T12:35:33.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:33.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2021-05-12T12:35:33.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:33.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2021-05-12T12:35:34.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:34.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2021-05-12T12:35:35.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:35.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2021-05-12T12:35:35.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T12:35:35.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2021-05-12T12:35:36.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2021-05-12T12:35:36.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2021-05-12T12:35:36.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2021-05-12T12:35:36.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local poolname=test1 2021-05-12T12:35:36.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local objectname=obj1-p1 2021-05-12T12:35:36.051 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: get_primary test1 obj1-p1 2021-05-12T12:35:36.052 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local poolname=test1 2021-05-12T12:35:36.052 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: local objectname=obj1-p1 2021-05-12T12:35:36.052 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:35:36.052 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: jq .acting_primary 2021-05-12T12:35:36.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local primary=1 2021-05-12T12:35:36.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:35:36.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1173: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-05-12T12:35:36.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2021-05-12T12:35:36.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:37.131 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:37.131 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:32:17.785586+0000 36'600 36:641 [1] 1 [1] 1 0'0 2021-05-12T12:31:59.477756+0000 0'0 2021-05-12T12:31:59.477756+0000 0 2021-05-12T12:35:37.131 INFO:tasks.workunit.client.0.smithi090.stdout:2.0 199 0 0 0 0 2084569088 0 0 596 596 active+clean 2021-05-12T12:32:19.437071+0000 36'596 36:634 [0] 0 [0] 0 0'0 2021-05-12T12:32:01.862272+0000 0'0 2021-05-12T12:32:01.862272+0000 0 2021-05-12T12:35:37.131 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:35:37.132 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:37.132 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:37.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2021-05-12T12:35:37.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T12:35:37.809 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 2 2021-05-12T12:35:37.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T12:35:38.671 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 2 size to 2 2021-05-12T12:35:38.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:39.034 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:39.035 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:32:17.785586+0000 36'600 36:641 [1] 1 [1] 1 0'0 2021-05-12T12:31:59.477756+0000 0'0 2021-05-12T12:31:59.477756+0000 0 2021-05-12T12:35:39.035 INFO:tasks.workunit.client.0.smithi090.stdout:2.0 199 0 0 0 0 2084569088 0 0 596 596 active+clean 2021-05-12T12:32:19.437071+0000 36'596 36:634 [0] 0 [0] 0 0'0 2021-05-12T12:32:01.862272+0000 0'0 2021-05-12T12:32:01.862272+0000 0 2021-05-12T12:35:39.035 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:35:39.035 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:39.037 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:39.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: set -o pipefail 2021-05-12T12:35:39.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: count=0 2021-05-12T12:35:39.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:39.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:39.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:39.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:39.389 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:39.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T12:35:39.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T12:35:41.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 0 -eq 10 2021-05-12T12:35:41.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 0 + 1 2021-05-12T12:35:41.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=1 2021-05-12T12:35:41.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:41.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:41.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:41.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:41.746 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:41.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 1 == 2 2021-05-12T12:35:41.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T12:35:43.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 1 -eq 10 2021-05-12T12:35:43.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T12:35:43.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=2 2021-05-12T12:35:43.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:43.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:43.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:43.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:44.104 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:44.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 1 == 2 2021-05-12T12:35:44.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T12:35:46.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 2 -eq 10 2021-05-12T12:35:46.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 2 + 1 2021-05-12T12:35:46.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=3 2021-05-12T12:35:46.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T12:35:46.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T12:35:46.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T12:35:46.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T12:35:46.465 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:46.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 2 == 2 2021-05-12T12:35:46.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:440: TEST_recovery_pool_priority: break 2021-05-12T12:35:46.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:450: TEST_recovery_pool_priority: set +o pipefail 2021-05-12T12:35:46.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T12:35:46.817 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:35:46.817 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 199 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:35:39.804605+0000 36'600 42:655 [1,0] 1 [1] 1 0'0 2021-05-12T12:31:59.477756+0000 0'0 2021-05-12T12:31:59.477756+0000 0 2021-05-12T12:35:46.818 INFO:tasks.workunit.client.0.smithi090.stdout:2.0 200 0 198 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:35:39.696440+0000 36'600 42:655 [0,1] 0 [0] 0 0'0 2021-05-12T12:32:01.862272+0000 0'0 2021-05-12T12:32:01.862272+0000 0 2021-05-12T12:35:46.818 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:35:46.818 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:35:46.819 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:35:46.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: get_asok_path osd.1 2021-05-12T12:35:46.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:35:46.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:35:46.831 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:35:46.831 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:35:46.831 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:35:46.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:35:46.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T12:35:46.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:35:46.931 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 2021-05-12T12:35:46.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: echo osd.1 2021-05-12T12:35:46.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:455: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:46.933 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:35:46.933 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:35:46.933 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:35:46.933 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:35:46.933 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:35:46.934 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:35:46.934 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:35:46.934 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:35:46.934 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 191, 2021-05-12T12:35:46.935 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:35:46.935 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:46.935 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:35:46.935 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:35:46.935 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:35:46.936 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:35:46.936 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:35:46.936 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:35:46.936 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:35:46.936 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:35:46.937 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "2.0", 2021-05-12T12:35:46.937 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 192, 2021-05-12T12:35:46.937 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:35:46.937 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:46.938 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:35:46.938 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:46.938 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:35:46.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.0 2021-05-12T12:35:46.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T12:35:46.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T12:35:46.941 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:35:46.941 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:35:46.942 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:35:46.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.0.asok 2021-05-12T12:35:46.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T12:35:46.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.0.asok dump_recovery_reservations 2021-05-12T12:35:47.035 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 2021-05-12T12:35:47.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.0 2021-05-12T12:35:47.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:47.037 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:35:47.037 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:35:47.037 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:35:47.037 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:35:47.038 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:35:47.038 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:35:47.038 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:35:47.038 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "2.0", 2021-05-12T12:35:47.039 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 192, 2021-05-12T12:35:47.039 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:35:47.039 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:47.039 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:35:47.039 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:35:47.040 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:35:47.040 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:35:47.040 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:35:47.040 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:35:47.041 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:35:47.041 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:35:47.041 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:35:47.041 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 191, 2021-05-12T12:35:47.041 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:35:47.042 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:47.042 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:35:47.042 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:35:47.042 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:35:47.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:47.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:35:47.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T12:35:47.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T12:35:47.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:462: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T12:35:47.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:47.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T12:35:47.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: PRIO=191 2021-05-12T12:35:47.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T12:35:47.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:47.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T12:35:47.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T12:35:47.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T12:35:47.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:477: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T12:35:47.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:47.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T12:35:47.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: PRIO=191 2021-05-12T12:35:47.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T12:35:47.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:47.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T12:35:47.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:492: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T12:35:47.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T12:35:47.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T12:35:47.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: PRIO=192 2021-05-12T12:35:47.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T12:35:47.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:47.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T12:35:47.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T12:35:47.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T12:35:47.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T12:35:47.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T12:35:47.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T12:35:47.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: PRIO=192 2021-05-12T12:35:47.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:513: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T12:35:47.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:520: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T12:35:47.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:35:47.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:35:47.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:35:47.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:35:47.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:35:47.066 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:35:47.066 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:35:47.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:35:47.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:35:47.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:35:47.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:35:47.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:35:47.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:35:47.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:35:47.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:35:47.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:35:47.517 INFO:tasks.workunit.client.0.smithi090.stderr:1' 2021-05-12T12:35:47.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:35:47.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:35:47.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:35:47.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836534 2021-05-12T12:35:47.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836534 2021-05-12T12:35:47.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836534' 2021-05-12T12:35:47.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:35:47.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:35:47.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949673012 2021-05-12T12:35:47.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949673012 2021-05-12T12:35:47.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2021-05-12T12:35:47.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:35:47.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836534 2021-05-12T12:35:47.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:35:47.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:35:47.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836534 2021-05-12T12:35:47.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:35:47.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836534 2021-05-12T12:35:47.805 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836534 2021-05-12T12:35:47.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2021-05-12T12:35:47.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:35:48.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836532 -lt 21474836534 2021-05-12T12:35:48.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:35:49.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:35:49.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:35:49.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836532 -lt 21474836534 2021-05-12T12:35:49.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:35:50.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 299 -eq 0 ']' 2021-05-12T12:35:50.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:35:50.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836534 -lt 21474836534 2021-05-12T12:35:50.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:35:50.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673012 2021-05-12T12:35:50.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:35:50.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:35:50.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949673012 2021-05-12T12:35:50.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:35:50.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949673012 2021-05-12T12:35:50.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2021-05-12T12:35:50.913 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 42949673012 2021-05-12T12:35:50.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:35:51.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949673012 -lt 42949673012 2021-05-12T12:35:51.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:35:51.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:51.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:51.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T12:35:51.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:51.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:51.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:51.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:51.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:51.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:51.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:52.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:52.154 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:52.154 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:52.154 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:52.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:52.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:35:52.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:35:52.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:35:52.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:52.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:52.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:52.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:52.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:52.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:52.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:52.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:52.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:52.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:53.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:53.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:53.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:53.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:53.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:53.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:53.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368235 2021-05-12T12:35:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368235 '!=' null 2021-05-12T12:35:54.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:54.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:54.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:54.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:54.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:54.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:54.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:54.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:54.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:54.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:54.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:54.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:54.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:54.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:54.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:55.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:55.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:55.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:55.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:55.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:55.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:55.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:55.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:55.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:55.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368235 2021-05-12T12:35:55.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368235 '!=' null 2021-05-12T12:35:55.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:55.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:55.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:55.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:55.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:55.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:55.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:55.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:55.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:55.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:55.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:56.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:56.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:56.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:56.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:56.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:56.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:56.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:56.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:56.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:56.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:56.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:56.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:56.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:56.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989230 2021-05-12T12:35:56.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989230 '!=' null 2021-05-12T12:35:56.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:56.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:56.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:57.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:57.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:57.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:57.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:57.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:57.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:57.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:57.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:57.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:57.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:57.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:58.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368228 2021-05-12T12:35:58.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368228 '!=' null 2021-05-12T12:35:58.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:58.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:58.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:58.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:58.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:58.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:58.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:58.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:58.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:58.520 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:58.520 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:35:58.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:35:58.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:35:58.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:35:58.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:35:59.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:35:59.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:35:59.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:35:59.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:35:59.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:35:59.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:35:59.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:35:59.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:35:59.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:35:59.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368176 2021-05-12T12:35:59.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368176 '!=' null 2021-05-12T12:35:59.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:35:59.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:35:59.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:35:59.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:35:59.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:35:59.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:35:59.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:35:59.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:35:59.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:35:59.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:35:59.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:00.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:00.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:00.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:00.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:00.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:00.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:00.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:00.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:00.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:00.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:00.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:00.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:00.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368176 2021-05-12T12:36:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368176 '!=' null 2021-05-12T12:36:01.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:01.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:01.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:01.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:01.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:01.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:01.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:01.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:01.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:01.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:01.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:01.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:01.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:01.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:02.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:02.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:02.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:02.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:02.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:02.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:02.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:02.367 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:02.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:02.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989131 2021-05-12T12:36:02.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989131 '!=' null 2021-05-12T12:36:02.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:02.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:02.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:02.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:02.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:02.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:02.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:02.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:02.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:02.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:02.955 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:03.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:03.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:03.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:03.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:03.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:03.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:03.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:03.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:03.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:03.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:03.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:03.792 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:03.792 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:04.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241847 2021-05-12T12:36:04.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241847 '!=' null 2021-05-12T12:36:04.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:04.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:04.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:04.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:04.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:04.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:04.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:04.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:04.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:04.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:04.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:04.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:04.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:04.794 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:04.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:05.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:05.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:05.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:05.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:05.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:05.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:05.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:05.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:05.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:05.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989165 2021-05-12T12:36:05.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989165 '!=' null 2021-05-12T12:36:05.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:05.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:05.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:05.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:05.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:05.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:05.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:05.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:05.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:05.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:05.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:06.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:06.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:06.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:06.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:06.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:06.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:06.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:06.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:06.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:06.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:06.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:06.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:06.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:07.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989165 2021-05-12T12:36:07.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989165 '!=' null 2021-05-12T12:36:07.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:07.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:07.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:07.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:07.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:07.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:07.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:07.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:07.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:07.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:07.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:07.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:07.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:07.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:07.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:08.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:08.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:08.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:08.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:08.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:08.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:08.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:08.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:08.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:08.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368243 2021-05-12T12:36:08.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368243 '!=' null 2021-05-12T12:36:08.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:08.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:08.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:08.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:08.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:08.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:08.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:08.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:08.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:08.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:08.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:09.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:09.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:09.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:09.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:09.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:09.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:09.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:09.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:09.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:09.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:09.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:09.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:09.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:10.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368250 2021-05-12T12:36:10.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368250 '!=' null 2021-05-12T12:36:10.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:10.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:10.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:10.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:10.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:10.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:10.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:10.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:10.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:10.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:10.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:10.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:10.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:10.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:10.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:11.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:11.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:11.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:11.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:11.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:11.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:11.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:11.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:11.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:11.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368250 2021-05-12T12:36:11.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368250 '!=' null 2021-05-12T12:36:11.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:11.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:11.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:11.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:11.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:11.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:11.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:11.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:11.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:11.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:11.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:12.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:12.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:12.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:12.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:12.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:12.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:12.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:12.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:12.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:12.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:12.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:12.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:12.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:13.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989272 2021-05-12T12:36:13.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989272 '!=' null 2021-05-12T12:36:13.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:13.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:13.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:13.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:13.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:13.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:13.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:13.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:13.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:13.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:13.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:13.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:13.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:13.642 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:13.642 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:14.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:14.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:14.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:14.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:14.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:14.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:14.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:14.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:14.128 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:14.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T12:36:14.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T12:36:14.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:14.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:14.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:14.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:14.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:14.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:14.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:14.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:14.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:14.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:14.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:15.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:15.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:15.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:15.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:15.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:15.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:15.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:15.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:15.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:16.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863016 2021-05-12T12:36:16.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863016 '!=' null 2021-05-12T12:36:16.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:16.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:16.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:16.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:16.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:16.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:16.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:16.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:16.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:16.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:16.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:16.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:16.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:16.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:16.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:16.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:16.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:16.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:16.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:16.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:16.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:16.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:16.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:16.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:17.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863016 2021-05-12T12:36:17.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863016 '!=' null 2021-05-12T12:36:17.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:17.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:17.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:17.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:17.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:17.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:17.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:17.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:17.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:17.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:17.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:18.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:18.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:18.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:18.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:18.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:18.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:18.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:18.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:18.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:18.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:18.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:18.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:18.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:18.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115707 2021-05-12T12:36:18.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115707 '!=' null 2021-05-12T12:36:18.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:19.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:19.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:19.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:19.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:19.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:19.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:19.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:19.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:19.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:19.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:19.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:19.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:19.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:19.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:19.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:19.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:19.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:19.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:19.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:19.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:19.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:19.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:19.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:20.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115674 2021-05-12T12:36:20.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115674 '!=' null 2021-05-12T12:36:20.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:20.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:20.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:20.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:20.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:20.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:20.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:20.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:20.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:20.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:20.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:20.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:20.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:20.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:20.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:21.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:21.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:21.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:21.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:21.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:21.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:21.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:21.367 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:21.367 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:21.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736752 2021-05-12T12:36:21.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736752 '!=' null 2021-05-12T12:36:21.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:21.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:21.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:21.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:21.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:21.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:21.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:21.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:21.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:21.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:21.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:22.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:22.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:22.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:22.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:22.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:22.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:22.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:22.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:22.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:22.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:22.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:22.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:22.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:23.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736752 2021-05-12T12:36:23.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736752 '!=' null 2021-05-12T12:36:23.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:23.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:23.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:23.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:23.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:23.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:23.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:23.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:23.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:23.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:23.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:23.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:24.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:24.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:24.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:24.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:24.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:24.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:24.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:24.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:24.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:24.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115701 2021-05-12T12:36:24.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115701 '!=' null 2021-05-12T12:36:24.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:24.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:24.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:24.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:24.801 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:24.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:24.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:24.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:24.802 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:24.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:24.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:25.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:25.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:25.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:25.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:25.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:25.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:25.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:25.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:25.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:25.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:25.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:25.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:25.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:26.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863049 2021-05-12T12:36:26.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863049 '!=' null 2021-05-12T12:36:26.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:26.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:26.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:26.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:26.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:26.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:26.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:26.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:26.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:26.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:26.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:26.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:26.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:26.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:26.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:27.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:27.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:27.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:27.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:27.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:27.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:27.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:27.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:27.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:27.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863049 2021-05-12T12:36:27.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863049 '!=' null 2021-05-12T12:36:27.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:27.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:27.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:27.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:27.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:27.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:27.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:27.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:27.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:27.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:27.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:28.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:28.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:28.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:28.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:28.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:28.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:28.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:28.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:28.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:28.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:28.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:28.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:28.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:28.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368350 2021-05-12T12:36:29.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368350 '!=' null 2021-05-12T12:36:29.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:29.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:29.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:29.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:29.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:29.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:29.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:29.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:29.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:29.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:29.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:29.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:29.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:29.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:29.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:29.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:29.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:29.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:30.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368346 2021-05-12T12:36:30.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368346 '!=' null 2021-05-12T12:36:30.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:30.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:30.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:30.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:30.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:30.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:30.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:30.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:30.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:30.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:30.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:31.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:31.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:31.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:31.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:31.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:31.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:31.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:31.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:31.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:31.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:31.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:31.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:31.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:32.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989397 2021-05-12T12:36:32.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989397 '!=' null 2021-05-12T12:36:32.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:32.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:32.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:32.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:32.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:32.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:32.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:32.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:32.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:32.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:32.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:32.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:32.496 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:32.496 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:32.496 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:32.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:32.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:32.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:32.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:32.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:32.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:32.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:32.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989397 2021-05-12T12:36:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989397 '!=' null 2021-05-12T12:36:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:33.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:33.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:33.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:33.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:33.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:33.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:33.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:33.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:33.921 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:33.921 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:33.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:34.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:34.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:34.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:34.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:34.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:34.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:36:34.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:36:34.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:34.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:35.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:35.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:35.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:35.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:35.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:35.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:35.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:35.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:35.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:35.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:35.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:35.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:35.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:35.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:35.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:35.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:35.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:35.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:35.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:35.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:35.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:36.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115690 2021-05-12T12:36:36.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115690 '!=' null 2021-05-12T12:36:36.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:36.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:36.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:36.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:36.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:36.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:36.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:36.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:36.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:36.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:36.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:36.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:36.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:36.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:37.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:37.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:37.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:37.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:37.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:37.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:37.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:37.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:37.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:37.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T12:36:37.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T12:36:37.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:37.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:37.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:38.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:38.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:38.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:38.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:38.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:38.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:38.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:38.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:38.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:38.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:38.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:38.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:38.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:38.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:38.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:38.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:38.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:38.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:38.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:38.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:38.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:39.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T12:36:39.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T12:36:39.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:39.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:39.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:39.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:39.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:39.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:39.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:39.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:39.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:39.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:39.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:39.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:39.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:39.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:40.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:40.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:40.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:40.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:40.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:40.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:40.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:40.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:40.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:40.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368296 2021-05-12T12:36:40.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368296 '!=' null 2021-05-12T12:36:40.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:40.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:40.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:40.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:40.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:40.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:40.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:40.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:40.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:40.882 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:40.882 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:41.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:41.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:41.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:41.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:41.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:41.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:41.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:41.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:41.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:41.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:41.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:41.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:41.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:42.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989259 2021-05-12T12:36:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989259 '!=' null 2021-05-12T12:36:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:42.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:42.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:42.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:42.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:42.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:42.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:42.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:42.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:42.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:42.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:42.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:42.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:43.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:43.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:43.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:43.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:43.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:43.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:43.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:43.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:43.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:43.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989259 2021-05-12T12:36:43.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989259 '!=' null 2021-05-12T12:36:43.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:43.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:43.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:43.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:43.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:43.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:43.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:43.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:43.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:43.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:43.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:44.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:44.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:44.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:44.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:44.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:44.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:44.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:44.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:44.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:44.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:44.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:44.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:44.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:45.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T12:36:45.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T12:36:45.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:45.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:45.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:45.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:45.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:45.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:45.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:45.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:45.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:45.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:45.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:45.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:45.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:45.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:45.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:46.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:46.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:46.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:46.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:46.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:46.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:46.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:46.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:46.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:46.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989329 2021-05-12T12:36:46.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989329 '!=' null 2021-05-12T12:36:46.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:46.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:46.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:46.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:46.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:46.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:46.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:46.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:46.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:46.699 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:46.699 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:47.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:47.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:47.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:47.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:47.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:47.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:47.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:47.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:47.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:47.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:47.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:47.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:47.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:48.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241958 2021-05-12T12:36:48.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241958 '!=' null 2021-05-12T12:36:48.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:48.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:48.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:48.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:48.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:48.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:48.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:48.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:48.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:48.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:48.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:48.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:48.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:48.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:48.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:48.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:48.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:48.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:48.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:48.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:48.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:48.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:48.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:48.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:49.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241958 2021-05-12T12:36:49.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241958 '!=' null 2021-05-12T12:36:49.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:49.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:49.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:49.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:49.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:49.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:49.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:49.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:49.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:49.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:49.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:49.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:49.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:49.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:49.910 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:50.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:50.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:50.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:50.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:50.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:50.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:50.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:50.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:50.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241981 2021-05-12T12:36:50.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241981 '!=' null 2021-05-12T12:36:50.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:50.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:50.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:51.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:51.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:51.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:51.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:51.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:51.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:51.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:51.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:51.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:51.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:51.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:51.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:51.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:51.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:51.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:51.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:51.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:51.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:51.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:51.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:51.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:52.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863024 2021-05-12T12:36:52.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863024 '!=' null 2021-05-12T12:36:52.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:52.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:52.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:52.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:52.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:52.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:52.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:52.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:52.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:52.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:52.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:52.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:52.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:52.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:52.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:53.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:53.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:53.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:53.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:53.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:53.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:53.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:53.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:53.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:53.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242028 2021-05-12T12:36:53.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242028 '!=' null 2021-05-12T12:36:53.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:53.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:53.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:54.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:54.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:54.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:54.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:54.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:54.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:54.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:54.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:54.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:54.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:54.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:54.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:54.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:54.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:54.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:54.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:54.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:54.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:54.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:55.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242028 2021-05-12T12:36:55.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242028 '!=' null 2021-05-12T12:36:55.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:55.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:55.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:55.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:55.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:55.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:55.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:55.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:55.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:55.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:55.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:55.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:55.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:55.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:55.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:56.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:56.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:56.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:56.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:56.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:56.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:56.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:56.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:56.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:56.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736725 2021-05-12T12:36:56.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736725 '!=' null 2021-05-12T12:36:56.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:56.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:56.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:57.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:57.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:57.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:57.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:57.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:57.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:57.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:57.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:57.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:57.353 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:57.353 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:57.353 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:57.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:57.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:57.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:57.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:57.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:57.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:57.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:57.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:57.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:58.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115692 2021-05-12T12:36:58.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115692 '!=' null 2021-05-12T12:36:58.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:58.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:58.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:58.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:58.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:58.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:58.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:36:58.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:36:58.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:36:58.453 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:36:58.453 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:36:58.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:36:58.827 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:36:58.827 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:36:58.827 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:36:59.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:36:59.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:36:59.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:36:59.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:36:59.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:36:59.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:36:59.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:36:59.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:36:59.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:36:59.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115727 2021-05-12T12:36:59.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115727 '!=' null 2021-05-12T12:36:59.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:36:59.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:36:59.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:36:59.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:36:59.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:36:59.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:36:59.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:00.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:00.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:00.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:00.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:00.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:00.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:00.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:00.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:00.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:00.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:00.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:01.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115727 2021-05-12T12:37:01.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115727 '!=' null 2021-05-12T12:37:01.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:01.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:01.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:01.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:01.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:01.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:01.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:01.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:01.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:01.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:01.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:01.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:01.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:01.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:02.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:02.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:02.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:02.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:02.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:02.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:02.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:02.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:02.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:03.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863049 2021-05-12T12:37:03.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863049 '!=' null 2021-05-12T12:37:03.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:03.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:03.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:03.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:03.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:03.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:03.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:03.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:03.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:03.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:03.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:03.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:03.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:03.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:03.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:04.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:04.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:04.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:04.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:04.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:04.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:04.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:04.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:04.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:04.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242041 2021-05-12T12:37:04.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242041 '!=' null 2021-05-12T12:37:04.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:04.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:04.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:04.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:04.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:04.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:04.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:04.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:04.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:04.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:04.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:05.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:05.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:05.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:05.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:05.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:05.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:05.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:05.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:05.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:05.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:05.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:05.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:05.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:06.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863075 2021-05-12T12:37:06.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863075 '!=' null 2021-05-12T12:37:06.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:06.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:06.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:06.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:06.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:06.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:06.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:06.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:06.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:06.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:06.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:06.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:06.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:06.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:06.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:07.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:07.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:07.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:07.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:07.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:07.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:07.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:07.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:07.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:07.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863075 2021-05-12T12:37:07.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863075 '!=' null 2021-05-12T12:37:07.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:07.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:07.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:07.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:07.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:07.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:07.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:07.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:07.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:07.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:07.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:08.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:08.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:08.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:08.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:08.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:08.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:08.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:08.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:08.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:08.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:08.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:08.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:08.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:08.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368359 2021-05-12T12:37:08.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368359 '!=' null 2021-05-12T12:37:08.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:08.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:08.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:09.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:09.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:09.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:09.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:09.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:09.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:09.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:09.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:09.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:09.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:09.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:09.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:09.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:09.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:09.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:09.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:09.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:09.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368336 2021-05-12T12:37:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368336 '!=' null 2021-05-12T12:37:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:10.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:10.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:10.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:10.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:10.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:10.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:10.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:10.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:10.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:10.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:10.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:10.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:11.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:11.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:11.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:11.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:11.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:11.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:11.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:11.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:11.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:11.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115623 2021-05-12T12:37:11.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115623 '!=' null 2021-05-12T12:37:11.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:11.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:11.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:12.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:12.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:12.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:12.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:12.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:12.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:12.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:12.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:12.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:12.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:12.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:12.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:12.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:12.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:12.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:12.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:12.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:12.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:12.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:12.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:12.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:13.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115623 2021-05-12T12:37:13.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115623 '!=' null 2021-05-12T12:37:13.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:13.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:13.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:13.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:13.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:13.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:13.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:13.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:13.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:13.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:13.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:13.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:13.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:13.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:13.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:14.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:14.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:14.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:14.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:14.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:14.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:14.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:14.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:14.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:14.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368279 2021-05-12T12:37:14.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368279 '!=' null 2021-05-12T12:37:14.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:14.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:14.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:14.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:14.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:14.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:14.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:14.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:14.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:14.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:14.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:15.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:15.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:15.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:15.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:15.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:15.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:15.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:15.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:15.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:15.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:15.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:15.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:15.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:16.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989222 2021-05-12T12:37:16.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989222 '!=' null 2021-05-12T12:37:16.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:16.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:16.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:16.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:16.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:16.396 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:16.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:16.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:16.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:16.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:16.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:16.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:16.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:16.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:16.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:17.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:17.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:17.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:17.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:17.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:17.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:17.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:17.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:17.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:17.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989222 2021-05-12T12:37:17.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989222 '!=' null 2021-05-12T12:37:17.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:17.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:17.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:17.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:17.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:17.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:17.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:17.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:17.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:17.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:17.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:18.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:18.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:18.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:18.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:18.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:18.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:18.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:18.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:18.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:18.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:18.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:18.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:19.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368234 2021-05-12T12:37:19.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368234 '!=' null 2021-05-12T12:37:19.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:19.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:19.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:19.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:19.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:19.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:19.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:19.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:19.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:19.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:19.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:19.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:19.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:19.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:19.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:20.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:20.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:20.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:20.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:20.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:20.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:20.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368223 2021-05-12T12:37:20.666 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368223 '!=' null 2021-05-12T12:37:20.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:20.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:20.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:20.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:20.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:20.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:20.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:20.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:20.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:20.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:20.770 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:21.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:21.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:21.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:21.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:21.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:21.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:21.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:21.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:21.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:21.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:21.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:21.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:21.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:22.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989219 2021-05-12T12:37:22.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989219 '!=' null 2021-05-12T12:37:22.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:22.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:22.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:22.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:22.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:22.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:22.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:22.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:22.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:22.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:22.262 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:22.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:22.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:22.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:22.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:23.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:23.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:23.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:23.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:23.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:23.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:23.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989219 2021-05-12T12:37:23.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989219 '!=' null 2021-05-12T12:37:23.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:23.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:23.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:23.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:23.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:23.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:23.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:23.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:23.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:23.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:23.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:24.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:24.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:24.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:24.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:24.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:24.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:24.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:24.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:24.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:24.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:24.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:24.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:24.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:25.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241949 2021-05-12T12:37:25.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241949 '!=' null 2021-05-12T12:37:25.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:25.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:25.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:25.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:25.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:25.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:25.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:25.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:25.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:25.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:25.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:25.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:25.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:26.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:26.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:26.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:26.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:26.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:26.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:26.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:26.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:26.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:26.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862984 2021-05-12T12:37:26.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862984 '!=' null 2021-05-12T12:37:26.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:26.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:26.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:26.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:26.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:26.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:26.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:26.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:26.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:26.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:26.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:26.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:26.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:26.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:26.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:27.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:27.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:27.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:27.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:27.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:27.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:27.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:27.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:27.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:27.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242008 2021-05-12T12:37:27.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242008 '!=' null 2021-05-12T12:37:27.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:27.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:27.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:28.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:28.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:28.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:28.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:28.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:28.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:28.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:28.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:28.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:28.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:28.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:28.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:28.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:28.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:28.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:28.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:28.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:29.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242008 2021-05-12T12:37:29.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242008 '!=' null 2021-05-12T12:37:29.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:29.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:29.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:29.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:29.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:29.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:29.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:29.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:29.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:29.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:29.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:29.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:29.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:29.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:29.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:30.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:30.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:30.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:30.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:30.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:30.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:30.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:30.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:30.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:30.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242026 2021-05-12T12:37:30.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242026 '!=' null 2021-05-12T12:37:30.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:30.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:30.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:30.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:30.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:30.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:30.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:30.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:30.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:30.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:30.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:31.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:31.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:31.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:31.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:31.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:31.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:31.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:31.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:31.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:31.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:31.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:31.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:31.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:32.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989365 2021-05-12T12:37:32.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989365 '!=' null 2021-05-12T12:37:32.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:32.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:32.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:32.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:32.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:32.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:32.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:32.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:32.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:32.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:32.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:32.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:32.796 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:32.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:33.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:33.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:33.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:33.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:33.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:33.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:33.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:33.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:33.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:33.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989365 2021-05-12T12:37:33.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989365 '!=' null 2021-05-12T12:37:33.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:33.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:33.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:33.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:33.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:33.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:33.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:33.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:33.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:33.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:33.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:34.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:34.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:34.244 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:34.244 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:34.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:34.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:34.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:34.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:34.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:35.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T12:37:35.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T12:37:35.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:35.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:35.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:35.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:35.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:35.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:35.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:35.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:35.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:35.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:35.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:35.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:36.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:36.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:36.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:36.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:36.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:36.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:36.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:36.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:36.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:36.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862960 2021-05-12T12:37:36.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862960 '!=' null 2021-05-12T12:37:36.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:36.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:36.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:36.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:36.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:36.750 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:36.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:36.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:36.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:36.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:36.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:37.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:37.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:37.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:37.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:37.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:37.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:37.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:37.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:37.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:37.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:37.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:37.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:37.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:38.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241972 2021-05-12T12:37:38.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241972 '!=' null 2021-05-12T12:37:38.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:38.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:38.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:38.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:38.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:38.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:38.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:38.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:38.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:38.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:38.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:38.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:38.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:38.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:38.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:39.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:39.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:39.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:39.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:39.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:39.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:39.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:39.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:39.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:39.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241972 2021-05-12T12:37:39.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241972 '!=' null 2021-05-12T12:37:39.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:39.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:39.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:39.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:39.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:39.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:39.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:39.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:39.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:39.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:39.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:40.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:40.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:40.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:40.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:40.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:40.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:40.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:40.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:40.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:40.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:40.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:40.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:40.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:41.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241926 2021-05-12T12:37:41.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241926 '!=' null 2021-05-12T12:37:41.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:41.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:41.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:41.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:41.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:41.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:41.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:41.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:41.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:41.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:41.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:41.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:41.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:41.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:41.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:41.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:41.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:41.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:41.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:42.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862898 2021-05-12T12:37:42.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862898 '!=' null 2021-05-12T12:37:42.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:42.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:42.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:42.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:42.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:42.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:42.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:42.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:42.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:42.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:42.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:42.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:42.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:42.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:42.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:43.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:43.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:43.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:43.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:43.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:43.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:43.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:43.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:43.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:43.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115604 2021-05-12T12:37:43.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115604 '!=' null 2021-05-12T12:37:43.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:43.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:43.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:43.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:44.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:44.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:44.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:44.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:44.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:44.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:44.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:44.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:44.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:44.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:44.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:44.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:44.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:44.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:44.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:44.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:44.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:44.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:44.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:44.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:45.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115604 2021-05-12T12:37:45.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115604 '!=' null 2021-05-12T12:37:45.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:45.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:45.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:45.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:45.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:45.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:45.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:45.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:45.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:45.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:45.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:45.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:45.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:45.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:46.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:46.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:46.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:46.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:46.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:46.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:46.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:46.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:46.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:46.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862931 2021-05-12T12:37:46.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862931 '!=' null 2021-05-12T12:37:46.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:46.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:46.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:46.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:46.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:46.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:46.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:46.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:46.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:46.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:46.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:47.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:47.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:47.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:47.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:47.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:47.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:47.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:47.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:47.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:47.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:47.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:47.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:47.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:48.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T12:37:48.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T12:37:48.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:48.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:48.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:48.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:48.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:48.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:48.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:48.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:48.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:48.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:48.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:48.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:48.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:48.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:49.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:49.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:49.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:49.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:49.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:49.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:49.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:49.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:49.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:49.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T12:37:49.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T12:37:49.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:49.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:49.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:49.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:49.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:49.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:49.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:49.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:49.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:49.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:49.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:50.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:50.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:50.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:50.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:50.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:50.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:50.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:50.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:51.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368321 2021-05-12T12:37:51.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368321 '!=' null 2021-05-12T12:37:51.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:51.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:51.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:51.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:51.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:51.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:51.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:51.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:51.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:51.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:51.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:51.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:51.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:51.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:51.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:52.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:52.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:52.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:52.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:52.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:52.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:52.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989371 2021-05-12T12:37:52.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989371 '!=' null 2021-05-12T12:37:52.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:52.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:52.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:52.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:52.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:52.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:52.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:52.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:52.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:52.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:52.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:53.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:53.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:53.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:53.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:53.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:53.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:53.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:53.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:54.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T12:37:54.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T12:37:54.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:54.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:54.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:54.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:54.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:54.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:54.271 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:54.271 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:54.271 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:54.271 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:54.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:54.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:54.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:54.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:55.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:55.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:55.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:55.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:55.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:55.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T12:37:55.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T12:37:55.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:55.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:55.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:55.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:55.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:55.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:55.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:55.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:55.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:55.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:56.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:56.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:56.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:56.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:56.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:56.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:56.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:56.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:56.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:56.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:56.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:56.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:57.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989367 2021-05-12T12:37:57.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989367 '!=' null 2021-05-12T12:37:57.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:57.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:57.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:57.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:57.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:57.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:57.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:57.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:57.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:57.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:57.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:57.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:57.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:57.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:57.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:57.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:57.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:57.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:57.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:57.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:57.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:57.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:57.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:57.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:58.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242011 2021-05-12T12:37:58.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242011 '!=' null 2021-05-12T12:37:58.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:58.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:58.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:37:58.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:37:58.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:37:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:37:58.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:37:58.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:37:58.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:37:58.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:37:58.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:37:58.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:37:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:37:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:37:58.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:37:59.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:37:59.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:37:59.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:37:59.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:37:59.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:37:59.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:37:59.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:37:59.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:37:59.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:37:59.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242048 2021-05-12T12:37:59.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242048 '!=' null 2021-05-12T12:37:59.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:37:59.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:37:59.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:00.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:00.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:00.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:00.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:00.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:00.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:00.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:00.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:00.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:00.376 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:00.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:00.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:00.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:00.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:00.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:00.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:00.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:00.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:00.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:00.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:00.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:01.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242048 2021-05-12T12:38:01.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242048 '!=' null 2021-05-12T12:38:01.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:01.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:01.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:01.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:01.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:01.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:01.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:01.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:01.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:01.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:01.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:01.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:01.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:01.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:01.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:02.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:02.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:02.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:02.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:02.308 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:02.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:02.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:02.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:02.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:02.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989330 2021-05-12T12:38:02.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989330 '!=' null 2021-05-12T12:38:02.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:02.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:02.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:02.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:02.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:02.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:02.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:02.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:02.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:02.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:02.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:03.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:03.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:03.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:03.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:03.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:03.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:03.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:03.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:03.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:03.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:03.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:03.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:03.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:04.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368283 2021-05-12T12:38:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368283 '!=' null 2021-05-12T12:38:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:04.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:04.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:04.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:04.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:04.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:04.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:04.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:04.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:04.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:04.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:04.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:04.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:04.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:05.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:05.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:05.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:05.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:05.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:05.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:05.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:05.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:05.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:06.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989245 2021-05-12T12:38:06.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989245 '!=' null 2021-05-12T12:38:06.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:06.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:06.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:06.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:06.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:06.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:06.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:06.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:06.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:06.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:06.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:06.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:06.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:06.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:06.627 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:07.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:07.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:07.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:07.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:07.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:07.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:07.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:07.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:07.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989245 2021-05-12T12:38:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989245 '!=' null 2021-05-12T12:38:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:07.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:07.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:07.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:07.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:07.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:07.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:07.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:07.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:07.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:07.699 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:08.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:08.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:08.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:08.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:08.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:08.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:08.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:08.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:08.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:08.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:08.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:08.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:08.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:09.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T12:38:09.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T12:38:09.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:09.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:09.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:09.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:09.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:09.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:09.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:09.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:09.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:09.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:09.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:09.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:09.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:09.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:09.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:10.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:10.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:10.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:10.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:10.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:10.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:10.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:10.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:10.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:10.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:10.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:10.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:10.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:10.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:10.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:10.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:10.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:10.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:10.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:10.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:10.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:10.954 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:10.955 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:11.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:11.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:11.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:11.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:11.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:11.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:11.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:11.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:11.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:11.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:38:11.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:38:11.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:11.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:11.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:12.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:12.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:12.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:12.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:12.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:12.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:12.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:12.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:12.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:12.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:12.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:12.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:12.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:12.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:12.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:12.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:13.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:38:13.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:38:13.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:13.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:13.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:13.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:13.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:13.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:13.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:13.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:13.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:13.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:13.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:13.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:13.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:13.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:13.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:14.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:14.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:14.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:14.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:14.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:14.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:14.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:14.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:14.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:14.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T12:38:14.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T12:38:14.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:14.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:14.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:14.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:14.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:14.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:14.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:14.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:14.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:14.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:14.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:15.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:15.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:15.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:15.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:15.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:15.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:15.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:15.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:15.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:16.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736679 2021-05-12T12:38:16.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736679 '!=' null 2021-05-12T12:38:16.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:16.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:16.321 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:16.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:16.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:16.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:16.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:16.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:16.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:16.423 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:16.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:16.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:16.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:16.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:16.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:17.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:17.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:17.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:17.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:17.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:17.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:17.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:17.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:17.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:17.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736679 2021-05-12T12:38:17.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736679 '!=' null 2021-05-12T12:38:17.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:17.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:17.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:17.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:17.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:17.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:17.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:17.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:17.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:17.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:17.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:18.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:18.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:18.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:18.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:18.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:18.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:18.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:18.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:18.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:18.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:18.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:18.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:18.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:19.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115705 2021-05-12T12:38:19.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115705 '!=' null 2021-05-12T12:38:19.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:19.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:19.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:19.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:19.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:19.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:19.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:19.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:19.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:19.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:19.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:19.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:19.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:19.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:19.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:20.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:20.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:20.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:20.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:20.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:20.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:20.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:20.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:20.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:20.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115676 2021-05-12T12:38:20.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115676 '!=' null 2021-05-12T12:38:20.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:20.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:20.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:20.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:20.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:20.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:20.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:20.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:20.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:20.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:20.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:21.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:21.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:21.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:21.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:21.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:21.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:21.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:21.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:21.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:21.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:21.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:21.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:21.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:22.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863024 2021-05-12T12:38:22.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863024 '!=' null 2021-05-12T12:38:22.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:22.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:22.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:22.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:22.197 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:22.198 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:22.198 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:22.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:22.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:22.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:22.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:22.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:22.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:22.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:22.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:23.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:23.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:23.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:23.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:23.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:23.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:23.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:23.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:23.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:23.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863024 2021-05-12T12:38:23.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863024 '!=' null 2021-05-12T12:38:23.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:23.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:23.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:23.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:23.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:23.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:23.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:23.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:23.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:23.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:23.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:24.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:24.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:24.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:24.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:24.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:24.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:24.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:24.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:24.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:24.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:24.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:25.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241928 2021-05-12T12:38:25.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241928 '!=' null 2021-05-12T12:38:25.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:25.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:25.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:25.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:25.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:25.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:25.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:25.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:25.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:25.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:25.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:25.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:25.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:25.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:25.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:26.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:26.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:26.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:26.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:26.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:26.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862905 2021-05-12T12:38:26.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862905 '!=' null 2021-05-12T12:38:26.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:26.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:26.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:26.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:26.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:26.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:26.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:26.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:26.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:26.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:26.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:26.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:26.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:26.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:26.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:27.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:27.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:27.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:27.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:27.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:27.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:27.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:27.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:27.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:28.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368273 2021-05-12T12:38:28.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368273 '!=' null 2021-05-12T12:38:28.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:28.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:28.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:28.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:28.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:28.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:28.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:28.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:28.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:28.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:28.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:28.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:28.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:28.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:28.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:28.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:28.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:28.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:28.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:28.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:28.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:28.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:28.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:29.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368273 2021-05-12T12:38:29.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368273 '!=' null 2021-05-12T12:38:29.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:29.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:29.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:29.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:29.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:29.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:29.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:29.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:29.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:29.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:29.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:29.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:29.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:29.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:29.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:30.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:30.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:30.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:30.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:30.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:30.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:30.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:30.396 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:30.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:30.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368270 2021-05-12T12:38:30.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368270 '!=' null 2021-05-12T12:38:30.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:30.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:30.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:30.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:30.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:30.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:30.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:30.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:31.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:31.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:31.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:31.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:31.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:31.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:31.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:31.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:31.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:31.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:32.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115606 2021-05-12T12:38:32.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115606 '!=' null 2021-05-12T12:38:32.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:32.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:32.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:32.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:32.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:32.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:32.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:32.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:32.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:32.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:32.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:32.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:32.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:32.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:32.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:33.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:33.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:33.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:33.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:33.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:33.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:33.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:33.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:33.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:33.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368284 2021-05-12T12:38:33.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368284 '!=' null 2021-05-12T12:38:33.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:33.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:33.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:33.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:33.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:33.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:33.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:33.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:33.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:34.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:34.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:34.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:34.221 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:34.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:34.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:34.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:34.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:34.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:34.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:34.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:34.711 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:35.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368284 2021-05-12T12:38:35.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368284 '!=' null 2021-05-12T12:38:35.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:35.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:35.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:35.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:35.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:35.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:35.292 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:35.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:35.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:35.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:35.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:35.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:35.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:35.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:35.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:36.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:36.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:36.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:36.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:36.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:36.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989308 2021-05-12T12:38:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989308 '!=' null 2021-05-12T12:38:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:36.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:36.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:36.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:36.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:36.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:37.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:37.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:37.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:37.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:37.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:37.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:37.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:37.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:37.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:37.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:37.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:37.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:37.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:38.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368294 2021-05-12T12:38:38.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368294 '!=' null 2021-05-12T12:38:38.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:38.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:38.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:38.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:38.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:38.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:38.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:38.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:38.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:38.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:38.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:38.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:38.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:38.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:38.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:39.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:39.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:39.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:39.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:39.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:39.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:39.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:39.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368294 2021-05-12T12:38:39.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368294 '!=' null 2021-05-12T12:38:39.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:39.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:39.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:39.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:39.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:39.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:39.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:39.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:39.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:39.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:40.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:40.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:40.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:40.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:40.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:40.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:40.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:40.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:40.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:40.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:40.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:40.643 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:40.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:41.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368320 2021-05-12T12:38:41.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368320 '!=' null 2021-05-12T12:38:41.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:41.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:41.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:41.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:41.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:41.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:41.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:41.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:41.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:41.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:41.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:41.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:41.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:41.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:41.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:42.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:42.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:42.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:42.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:42.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:42.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:42.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:42.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:42.100 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:42.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989308 2021-05-12T12:38:42.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989308 '!=' null 2021-05-12T12:38:42.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:42.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:42.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:42.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:42.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:42.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:42.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:42.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:42.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:42.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:42.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:43.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:43.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:43.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:43.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:43.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:43.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:43.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:43.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:43.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:43.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:43.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:43.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:43.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:44.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241957 2021-05-12T12:38:44.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241957 '!=' null 2021-05-12T12:38:44.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:44.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:44.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:44.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:44.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:44.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:44.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:44.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:44.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:44.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:44.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:44.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:44.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:44.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:44.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:44.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:44.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:44.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:44.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:44.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:44.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:44.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:44.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:44.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:45.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241957 2021-05-12T12:38:45.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241957 '!=' null 2021-05-12T12:38:45.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:45.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:45.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:45.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:45.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:45.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:45.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:45.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:45.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:45.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:45.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:45.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:45.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:45.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:45.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:46.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:46.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:46.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:46.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:46.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:46.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:46.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:46.376 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:46.376 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:46.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862942 2021-05-12T12:38:46.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862942 '!=' null 2021-05-12T12:38:46.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:46.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:46.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:46.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:46.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:46.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:46.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:46.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:46.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:46.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:46.975 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:47.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:47.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:47.328 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:47.328 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:47.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:47.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:47.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:47.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:47.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:47.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:47.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:47.807 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:47.807 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:48.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242013 2021-05-12T12:38:48.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242013 '!=' null 2021-05-12T12:38:48.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:48.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:48.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:48.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:48.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:48.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:48.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:48.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:48.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:48.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:48.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:48.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:48.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:48.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:48.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:49.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:49.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:49.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:49.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:49.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:49.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:49.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:49.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:49.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:49.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241992 2021-05-12T12:38:49.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241992 '!=' null 2021-05-12T12:38:49.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:49.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:49.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:49.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:49.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:49.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:49.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:49.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:49.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:49.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:49.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:50.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:50.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:50.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:50.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:50.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:50.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:50.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:50.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:50.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:50.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:50.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:50.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:50.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:51.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241992 2021-05-12T12:38:51.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241992 '!=' null 2021-05-12T12:38:51.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:51.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:51.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:51.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:51.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:51.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:51.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:51.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:51.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:51.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:51.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:51.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:51.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:51.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:51.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:52.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:52.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:52.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:52.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:52.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:52.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:52.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:52.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:52.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:52.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862968 2021-05-12T12:38:52.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862968 '!=' null 2021-05-12T12:38:52.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:52.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:52.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:52.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:52.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:52.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:52.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:52.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:52.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:52.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:52.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:53.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:53.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:53.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:53.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:53.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:53.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:54.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241983 2021-05-12T12:38:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241983 '!=' null 2021-05-12T12:38:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:54.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:54.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:54.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:54.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:54.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:54.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:54.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:54.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:54.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:54.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:54.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:54.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:55.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:55.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:55.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:55.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:55.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863020 2021-05-12T12:38:55.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863020 '!=' null 2021-05-12T12:38:55.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:55.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:55.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:55.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:55.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:55.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:55.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:55.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:55.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:55.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:55.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:56.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:56.323 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:56.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:56.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:56.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:56.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:56.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:56.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:56.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:56.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:56.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:56.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:56.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:57.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863020 2021-05-12T12:38:57.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863020 '!=' null 2021-05-12T12:38:57.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:57.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:57.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:57.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:57.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:57.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:57.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:57.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:57.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:57.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:57.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:57.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:57.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:57.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:57.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:58.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:58.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:58.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:58.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:58.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:58.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:38:58.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241965 2021-05-12T12:38:58.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241965 '!=' null 2021-05-12T12:38:58.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:38:58.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:38:58.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:38:58.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:38:58.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:38:58.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:38:58.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:38:58.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:38:58.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:38:58.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:38:58.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:38:59.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:38:59.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:38:59.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:38:59.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:38:59.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:38:59.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:38:59.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:38:59.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:38:59.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:38:59.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:38:59.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:38:59.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:38:59.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:00.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241955 2021-05-12T12:39:00.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241955 '!=' null 2021-05-12T12:39:00.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:00.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:00.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:00.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:00.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:00.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:00.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:00.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:00.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:00.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:00.228 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:00.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:00.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:00.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:00.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:01.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:01.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:01.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:01.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:01.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:01.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:01.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:01.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:01.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:01.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241955 2021-05-12T12:39:01.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241955 '!=' null 2021-05-12T12:39:01.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:01.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:01.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:01.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:01.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:01.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:01.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:01.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:01.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:01.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:01.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:02.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:02.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:02.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:02.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:02.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:02.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:02.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:02.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:02.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:02.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:02.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:02.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:02.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:02.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863003 2021-05-12T12:39:02.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863003 '!=' null 2021-05-12T12:39:02.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:02.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:02.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:03.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:03.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:03.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:03.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:03.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:03.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:03.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:03.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:03.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:03.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:03.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:03.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:03.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:03.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:03.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:03.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:03.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:03.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:03.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:03.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:03.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:04.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T12:39:04.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T12:39:04.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:04.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:04.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:04.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:04.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:04.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:04.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:04.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:04.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:04.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:04.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:04.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:04.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:04.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:04.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:05.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:05.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:05.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:05.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:05.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:05.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:05.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:05.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:05.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:05.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T12:39:05.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T12:39:05.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:05.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:05.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:05.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:05.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:05.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:05.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:05.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:05.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:05.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:05.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:06.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:06.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:06.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:06.415 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:06.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:06.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:06.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:06.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:06.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:06.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:06.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:06.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:06.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:07.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989297 2021-05-12T12:39:07.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989297 '!=' null 2021-05-12T12:39:07.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:07.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:07.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:07.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:07.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:07.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:07.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:07.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:07.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:07.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:07.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:08.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:08.037 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:08.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:08.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:08.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:08.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:08.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:08.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:08.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:08.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:08.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:08.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:08.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:09.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368314 2021-05-12T12:39:09.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368314 '!=' null 2021-05-12T12:39:09.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:09.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:09.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:09.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:09.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:09.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:09.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:09.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:09.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:09.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:09.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:09.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:09.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:09.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:09.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:09.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:09.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:09.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:09.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:09.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:09.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:09.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:09.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:09.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:10.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368316 2021-05-12T12:39:10.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368316 '!=' null 2021-05-12T12:39:10.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:10.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:10.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:10.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:10.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:10.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:10.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:10.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:10.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:10.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:10.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:10.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:10.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:10.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:10.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:11.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:11.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:11.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:11.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:11.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:11.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:11.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:11.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:11.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:11.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989306 2021-05-12T12:39:11.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989306 '!=' null 2021-05-12T12:39:11.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:11.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:11.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:12.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:12.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:12.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:12.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:12.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:12.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:12.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:12.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:12.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:12.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:12.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:12.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:12.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:12.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:12.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:12.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:12.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:12.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:12.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:12.857 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:12.857 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:13.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989306 2021-05-12T12:39:13.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989306 '!=' null 2021-05-12T12:39:13.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:13.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:13.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:13.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:13.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:13.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:13.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:13.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:13.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:13.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:13.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:13.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:13.792 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:13.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:13.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:14.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:14.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:14.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:14.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:14.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:14.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:14.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:14.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:14.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:14.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368320 2021-05-12T12:39:14.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368320 '!=' null 2021-05-12T12:39:14.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:14.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:14.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:14.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:14.882 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:14.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:14.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:15.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:15.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:15.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:15.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:15.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:15.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:15.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:15.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:15.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:15.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:15.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:15.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:15.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:16.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115670 2021-05-12T12:39:16.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115670 '!=' null 2021-05-12T12:39:16.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:16.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:16.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:16.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:16.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:16.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:16.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:16.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:16.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:16.309 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:16.310 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:16.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:16.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:16.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:17.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:17.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:17.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:17.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:17.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:17.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:17.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:17.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:17.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:17.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115670 2021-05-12T12:39:17.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115670 '!=' null 2021-05-12T12:39:17.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:17.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:17.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:17.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:17.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:17.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:17.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:17.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:17.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:17.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:17.732 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:18.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:18.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:18.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:18.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:18.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:18.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:18.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:18.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:18.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:18.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:18.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:18.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:18.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:19.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368322 2021-05-12T12:39:19.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368322 '!=' null 2021-05-12T12:39:19.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:19.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:19.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:19.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:19.169 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:19.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:19.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:19.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:19.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:19.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:19.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:19.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:19.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:19.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:19.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:20.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:20.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:20.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:20.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:20.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:20.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:20.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:20.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:20.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:20.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368313 2021-05-12T12:39:20.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368313 '!=' null 2021-05-12T12:39:20.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:20.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:20.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:20.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:20.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:20.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:20.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:20.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:20.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:20.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:20.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:20.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:20.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:21.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:21.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:21.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:21.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:21.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:21.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:21.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:21.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:21.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:21.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989358 2021-05-12T12:39:21.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989358 '!=' null 2021-05-12T12:39:21.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:22.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:22.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:22.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:22.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:22.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:22.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:22.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:22.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:22.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:22.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:22.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:22.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:22.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:22.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:22.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:22.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:22.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:22.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:22.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:22.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:22.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:23.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989358 2021-05-12T12:39:23.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989358 '!=' null 2021-05-12T12:39:23.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:23.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:23.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:23.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:23.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:23.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:23.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:23.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:23.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:23.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:23.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:23.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:23.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:23.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:23.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:24.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:24.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:24.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:24.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:24.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:24.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:24.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:24.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:24.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:24.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368334 2021-05-12T12:39:24.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368334 '!=' null 2021-05-12T12:39:24.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:24.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:24.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:25.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:25.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:25.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:25.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:25.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:25.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:25.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:25.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:25.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:25.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:25.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:25.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:25.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:25.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:25.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:25.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:25.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:25.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:25.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:25.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:25.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:26.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989323 2021-05-12T12:39:26.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989323 '!=' null 2021-05-12T12:39:26.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:26.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:26.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:26.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:26.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:26.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:26.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:26.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:26.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:26.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:26.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:26.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:26.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:26.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:26.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:27.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:27.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:27.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:27.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:27.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:27.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:27.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:27.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:27.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:27.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241945 2021-05-12T12:39:27.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241945 '!=' null 2021-05-12T12:39:27.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:27.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:27.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:28.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:28.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:28.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:28.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:28.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:28.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:28.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:28.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:28.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:28.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:28.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:28.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:28.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:28.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:28.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:28.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:28.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:28.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:28.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:28.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:28.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:29.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241945 2021-05-12T12:39:29.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241945 '!=' null 2021-05-12T12:39:29.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:29.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:29.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:29.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:29.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:29.465 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:29.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:29.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:29.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:29.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:29.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:29.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:29.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:29.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:29.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:30.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:30.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:30.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:30.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:30.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:30.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:30.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:30.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:30.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:30.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241970 2021-05-12T12:39:30.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241970 '!=' null 2021-05-12T12:39:30.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:30.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:30.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:30.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:30.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:30.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:30.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:30.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:30.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:30.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:30.917 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:31.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:31.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:31.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:31.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:31.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:31.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:31.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:31.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:31.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:31.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:31.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:31.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:31.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:32.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736540 2021-05-12T12:39:32.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736540 '!=' null 2021-05-12T12:39:32.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:32.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:32.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:32.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:32.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:32.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:32.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:32.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:32.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:32.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:32.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:32.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:32.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:32.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:32.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:33.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:33.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:33.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:33.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:33.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:33.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:33.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:33.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736540 2021-05-12T12:39:33.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736540 '!=' null 2021-05-12T12:39:33.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:33.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:33.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:33.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:33.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:33.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:33.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:33.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:33.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:33.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:34.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:34.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:34.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:34.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:34.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:34.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:34.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:34.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:35.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115583 2021-05-12T12:39:35.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115583 '!=' null 2021-05-12T12:39:35.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:35.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:35.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:35.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:35.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:35.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:35.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:35.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:35.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:35.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:35.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:35.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:35.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:35.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:35.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:36.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:36.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:36.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:36.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:36.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:36.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:36.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:36.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862923 2021-05-12T12:39:36.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862923 '!=' null 2021-05-12T12:39:36.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:36.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:36.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:36.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:36.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:36.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:36.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:36.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:36.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:36.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:37.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:37.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:37.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:37.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:37.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:37.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:37.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:37.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:37.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:37.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:38.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241952 2021-05-12T12:39:38.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241952 '!=' null 2021-05-12T12:39:38.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:38.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:38.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:38.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:38.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:38.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:38.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:38.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:38.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:38.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:38.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:38.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:38.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:38.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:38.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:38.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:38.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:38.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:38.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:38.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:39.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241952 2021-05-12T12:39:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241952 '!=' null 2021-05-12T12:39:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:39.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:39.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:39.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:39.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:39.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:39.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:39.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:39.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:39.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:39.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:39.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:39.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:39.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:39.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:40.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:40.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:40.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:40.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:40.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:40.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:40.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:40.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:40.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:41.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241955 2021-05-12T12:39:41.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241955 '!=' null 2021-05-12T12:39:41.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:41.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:41.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:41.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:41.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:41.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:41.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:41.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:41.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:41.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:41.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:41.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:41.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:41.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:41.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:42.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:42.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:42.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:42.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:42.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:42.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:42.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:42.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:42.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:42.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862936 2021-05-12T12:39:42.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862936 '!=' null 2021-05-12T12:39:42.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:42.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:42.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:42.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:42.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:42.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:42.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:42.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:42.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:42.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:42.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:43.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:43.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:43.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:43.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:43.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:43.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:43.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:43.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:43.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:43.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:43.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:43.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:43.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:43.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368330 2021-05-12T12:39:43.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368330 '!=' null 2021-05-12T12:39:43.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:43.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:43.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:44.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:44.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:44.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:44.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:44.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:44.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:44.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:44.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:44.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:44.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:44.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:44.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:44.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:44.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:44.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:44.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:44.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:44.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:44.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:44.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:44.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:45.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368330 2021-05-12T12:39:45.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368330 '!=' null 2021-05-12T12:39:45.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:45.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:45.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:45.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:45.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:45.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:45.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:45.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:45.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:45.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:45.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:46.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:46.089 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:46.089 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:46.089 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:46.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:46.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:46.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:46.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:46.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:47.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115663 2021-05-12T12:39:47.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115663 '!=' null 2021-05-12T12:39:47.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:47.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:47.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:47.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:47.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:47.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:47.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:47.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:47.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:47.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:47.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:47.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:47.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:47.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:47.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:48.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:48.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:48.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:48.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:48.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:48.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:48.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:48.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:48.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:48.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368285 2021-05-12T12:39:48.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368285 '!=' null 2021-05-12T12:39:48.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:48.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:48.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:48.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:48.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:48.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:48.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:48.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:48.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:48.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:48.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:49.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:49.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:49.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:49.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:49.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:49.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:49.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:49.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:49.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:49.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:49.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:49.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:49.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:50.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368282 2021-05-12T12:39:50.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368282 '!=' null 2021-05-12T12:39:50.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:50.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:50.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:50.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:50.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:50.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:50.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:50.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:50.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:50.119 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:50.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:50.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:50.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:50.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:50.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:50.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:50.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:50.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:50.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:50.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:50.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:50.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:50.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:50.985 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:51.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368282 2021-05-12T12:39:51.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368282 '!=' null 2021-05-12T12:39:51.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:51.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:51.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:51.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:51.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:51.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:51.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:51.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:51.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:51.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:51.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:51.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:51.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:51.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:51.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:52.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:52.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:52.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:52.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:52.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:52.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989252 2021-05-12T12:39:52.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989252 '!=' null 2021-05-12T12:39:52.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:52.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:52.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:53.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:53.076 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:53.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:53.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:53.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:53.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:53.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:53.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:53.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:53.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:53.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:53.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:53.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:53.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:53.930 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:53.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368247 2021-05-12T12:39:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368247 '!=' null 2021-05-12T12:39:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:54.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:54.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:54.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:54.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:54.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:54.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:54.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:54.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:55.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:55.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:55.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:55.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:55.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:55.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:55.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:55.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:55.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:55.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:55.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:55.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:55.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:56.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989213 2021-05-12T12:39:56.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989213 '!=' null 2021-05-12T12:39:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:56.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:56.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:56.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:56.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:56.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:56.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:56.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:56.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:56.180 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:56.180 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:56.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:56.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:56.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:56.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:57.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:57.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:57.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:57.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:57.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:57.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:57.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:57.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:57.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:57.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989213 2021-05-12T12:39:57.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989213 '!=' null 2021-05-12T12:39:57.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:57.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:57.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:57.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:57.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:57.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:57.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:57.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:57.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:57.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:57.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:58.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:58.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:58.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:58.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:58.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:58.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:58.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:58.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:58.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:58.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:58.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:58.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:58.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:39:59.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241871 2021-05-12T12:39:59.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241871 '!=' null 2021-05-12T12:39:59.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:39:59.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:39:59.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:39:59.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:39:59.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:39:59.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:39:59.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:39:59.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:39:59.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:39:59.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:39:59.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:39:59.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:39:59.502 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:39:59.503 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:39:59.503 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:39:59.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:39:59.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:39:59.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:39:59.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:39:59.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:39:59.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:39:59.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:39:59.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:39:59.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:00.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241874 2021-05-12T12:40:00.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241874 '!=' null 2021-05-12T12:40:00.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:00.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:00.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:00.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:00.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:00.569 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:00.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:00.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:00.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:00.570 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:00.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:00.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:00.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:00.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:00.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:01.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:01.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:01.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:01.421 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:01.421 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:01.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862746 2021-05-12T12:40:01.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862746 '!=' null 2021-05-12T12:40:01.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:01.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:01.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:01.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:01.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:01.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:01.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:01.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:01.996 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:01.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:01.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:02.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:02.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:02.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:02.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:02.828 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:02.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:02.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:02.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:02.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:02.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:02.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:02.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:02.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:03.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862746 2021-05-12T12:40:03.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862746 '!=' null 2021-05-12T12:40:03.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:03.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:03.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:03.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:03.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:03.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:03.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:03.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:03.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:03.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:03.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:04.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:04.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:04.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:04.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:04.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:04.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:04.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:04.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:04.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:04.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:04.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:04.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:04.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:05.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241863 2021-05-12T12:40:05.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241863 '!=' null 2021-05-12T12:40:05.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:05.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:05.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:05.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:05.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:05.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:05.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:05.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:05.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:05.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:05.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:05.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:05.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:05.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:06.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:06.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:06.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:06.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:06.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:06.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:06.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:06.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:06.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:06.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862876 2021-05-12T12:40:06.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862876 '!=' null 2021-05-12T12:40:06.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:06.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:06.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:06.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:06.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:06.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:06.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:06.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:06.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:06.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:06.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:07.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:07.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:07.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:07.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:07.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:07.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:07.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:07.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:07.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:07.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:07.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:07.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:07.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:08.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241897 2021-05-12T12:40:08.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241897 '!=' null 2021-05-12T12:40:08.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:08.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:08.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:08.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:08.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:08.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:08.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:08.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:08.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:08.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:08.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:08.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:08.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:08.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:08.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:08.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:08.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:08.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:08.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:08.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:09.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241897 2021-05-12T12:40:09.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241897 '!=' null 2021-05-12T12:40:09.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:09.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:09.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:09.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:09.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:09.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:09.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:09.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:09.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:09.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:09.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:09.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:09.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:09.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:09.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:10.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:10.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:10.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:10.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:10.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T12:40:10.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T12:40:10.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:10.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:10.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:11.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:11.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:11.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:11.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:11.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:11.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:11.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:11.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:11.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:11.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:11.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:11.376 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:11.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:11.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:11.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:11.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:11.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:11.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:11.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:11.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:11.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:12.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862955 2021-05-12T12:40:12.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862955 '!=' null 2021-05-12T12:40:12.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:12.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:12.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:12.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:12.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:12.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:12.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:12.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:12.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:12.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:12.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:12.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:12.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:12.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:12.877 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:13.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:13.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:13.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:13.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:13.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:13.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:13.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:13.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:13.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:13.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241969 2021-05-12T12:40:13.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241969 '!=' null 2021-05-12T12:40:13.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:13.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:13.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:13.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:13.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:13.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:13.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:13.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:13.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:13.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:13.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:14.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:14.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:14.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:14.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:14.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:14.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:14.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:14.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:14.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:14.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:14.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:14.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:14.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:15.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241969 2021-05-12T12:40:15.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241969 '!=' null 2021-05-12T12:40:15.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:15.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:15.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:15.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:15.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:15.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:15.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:15.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:15.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:15.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:15.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:15.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:15.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:15.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:16.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:16.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:16.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:16.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:16.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:16.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989266 2021-05-12T12:40:16.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989266 '!=' null 2021-05-12T12:40:16.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:16.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:16.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:17.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:17.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:17.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:17.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:17.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:17.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:17.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:17.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:17.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:17.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:17.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:17.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:17.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:17.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:17.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:17.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:17.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:17.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:18.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368269 2021-05-12T12:40:18.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368269 '!=' null 2021-05-12T12:40:18.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:18.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:18.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:18.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:18.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:18.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:18.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:18.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:18.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:18.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:18.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:18.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:18.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:18.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:18.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:19.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:19.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:19.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:19.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:19.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:19.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:19.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:19.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:19.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:19.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T12:40:19.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T12:40:19.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:19.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:19.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:19.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:19.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:19.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:19.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:19.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:19.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:19.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:19.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:20.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:20.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:20.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:20.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:20.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:20.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:20.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:20.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:20.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:20.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:20.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:20.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:20.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:21.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T12:40:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T12:40:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:21.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:21.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:21.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:21.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:21.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:21.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:21.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:21.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:21.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:21.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:21.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:21.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:21.891 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:22.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:22.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:22.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:22.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:22.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:22.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:22.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989243 2021-05-12T12:40:22.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989243 '!=' null 2021-05-12T12:40:22.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:22.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:22.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:22.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:22.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:22.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:22.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:22.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:22.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:22.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:22.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:23.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:23.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:23.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:23.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:23.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:23.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:23.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:23.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:23.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:23.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:23.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:23.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:23.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:24.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368277 2021-05-12T12:40:24.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368277 '!=' null 2021-05-12T12:40:24.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:24.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:24.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:24.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:24.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:24.424 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:24.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:24.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:24.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:24.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:24.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:24.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:24.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:24.778 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:25.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:25.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:25.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:25.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:25.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:25.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:25.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:25.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:25.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:25.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368277 2021-05-12T12:40:25.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368277 '!=' null 2021-05-12T12:40:25.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:25.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:25.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:25.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:25.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:25.854 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:25.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:25.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:25.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:25.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:25.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:26.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:26.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:26.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:26.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:26.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:26.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:26.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:26.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:26.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:26.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:27.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989314 2021-05-12T12:40:27.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989314 '!=' null 2021-05-12T12:40:27.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:27.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:27.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:27.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:27.343 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:27.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:27.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:27.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:27.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:27.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:27.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:27.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:27.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:27.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:27.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:28.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:28.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:28.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:28.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:28.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:28.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:28.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:28.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:28.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:28.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242003 2021-05-12T12:40:28.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242003 '!=' null 2021-05-12T12:40:28.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:28.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:28.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:28.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:28.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:28.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:28.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:28.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:28.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:28.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:28.778 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:29.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:29.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:29.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:29.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:29.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:29.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:29.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:29.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:29.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:29.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:29.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:29.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:29.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:30.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242020 2021-05-12T12:40:30.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242020 '!=' null 2021-05-12T12:40:30.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:30.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:30.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:30.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:30.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:30.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:30.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:30.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:30.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:30.188 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:30.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:30.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:30.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:30.559 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:30.559 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:31.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:31.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:31.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:31.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:31.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:31.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:31.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:31.063 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:31.063 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:31.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242020 2021-05-12T12:40:31.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242020 '!=' null 2021-05-12T12:40:31.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:31.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:31.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:31.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:31.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:31.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:31.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:31.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:31.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:31.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:31.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:32.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:32.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:32.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:32.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:32.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:32.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:32.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:32.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:32.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:32.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:32.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:32.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:32.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:33.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989299 2021-05-12T12:40:33.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989299 '!=' null 2021-05-12T12:40:33.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:33.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:33.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:33.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:33.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:33.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:33.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:33.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:33.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:33.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:33.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:33.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:33.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:33.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:33.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:33.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:33.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:33.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:33.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:33.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:33.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:33.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:33.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:34.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T12:40:34.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T12:40:34.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:34.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:34.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:34.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:34.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:34.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:34.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:34.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:34.970 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:34.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:34.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:35.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:35.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:35.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:35.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:35.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:35.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:35.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:35.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:35.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:35.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989267 2021-05-12T12:40:35.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989267 '!=' null 2021-05-12T12:40:35.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:35.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:35.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:36.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:36.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:36.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:36.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:36.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:36.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:36.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:36.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:36.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:36.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:36.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:36.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:36.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:36.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:36.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:36.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:36.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:36.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:36.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:36.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:36.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:37.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989267 2021-05-12T12:40:37.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989267 '!=' null 2021-05-12T12:40:37.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:37.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:37.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:37.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:37.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:37.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:37.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:37.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:37.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:37.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:37.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:37.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:37.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:37.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:37.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:38.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:38.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:38.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:38.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:38.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:38.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:38.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:38.408 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:38.409 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:38.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368251 2021-05-12T12:40:38.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368251 '!=' null 2021-05-12T12:40:38.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:38.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:38.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:39.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:39.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:39.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:39.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:39.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:39.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:39.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:39.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:39.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:39.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:39.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:39.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:39.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:39.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:39.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:39.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:39.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:39.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:39.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:40.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:40:40.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:40:40.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:40.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:40.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:40.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:40.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:40.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:40.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:40.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:40.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:40.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:40.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:40.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:40.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:40.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:40.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:41.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:41.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:41.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:41.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:41.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:41.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:41.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:41.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:41.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:41.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862822 2021-05-12T12:40:41.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862822 '!=' null 2021-05-12T12:40:41.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:41.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:41.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:41.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:41.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:41.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:41.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:41.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:41.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:41.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:41.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:42.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:42.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:42.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:42.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:42.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:42.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:43.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862822 2021-05-12T12:40:43.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862822 '!=' null 2021-05-12T12:40:43.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:43.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:43.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:43.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:43.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:43.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:43.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:43.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:43.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:43.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:43.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:43.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:43.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:43.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:43.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:44.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:44.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:44.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:44.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:44.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:44.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:44.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:44.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:44.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:44.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115583 2021-05-12T12:40:44.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115583 '!=' null 2021-05-12T12:40:44.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:44.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:44.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:44.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:44.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:44.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:44.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:44.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:44.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:44.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:44.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:45.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:45.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:45.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:45.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:45.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:45.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:45.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:45.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:45.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:45.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:45.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:45.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:45.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:46.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736561 2021-05-12T12:40:46.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736561 '!=' null 2021-05-12T12:40:46.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:46.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:46.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:46.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:46.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:46.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:46.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:46.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:46.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:46.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:46.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:46.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:46.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:46.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:46.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:47.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:47.266 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:47.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:47.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:47.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:47.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:47.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:47.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:47.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:47.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736561 2021-05-12T12:40:47.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736561 '!=' null 2021-05-12T12:40:47.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:47.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:47.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:47.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:47.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:47.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:47.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:47.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:47.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:47.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:47.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:48.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:48.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:48.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:48.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:48.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:48.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:48.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:48.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:48.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:48.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:48.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:48.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:49.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115646 2021-05-12T12:40:49.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115646 '!=' null 2021-05-12T12:40:49.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:49.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:49.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:49.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:49.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:49.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:49.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:49.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:49.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:49.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:49.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:49.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:49.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:49.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:49.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:50.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:50.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:50.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:50.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:50.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:50.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:50.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:50.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:50.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:50.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115707 2021-05-12T12:40:50.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115707 '!=' null 2021-05-12T12:40:50.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:50.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:50.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:50.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:50.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:50.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:50.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:50.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:50.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:50.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:50.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:51.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:51.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:51.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:51.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:51.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:51.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:51.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:51.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:51.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:51.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:52.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863015 2021-05-12T12:40:52.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863015 '!=' null 2021-05-12T12:40:52.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:52.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:52.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:52.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:52.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:52.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:52.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:52.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:52.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:52.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:52.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:53.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:53.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:53.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:53.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:53.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:53.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:53.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:53.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:53.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:53.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863015 2021-05-12T12:40:53.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863015 '!=' null 2021-05-12T12:40:53.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:53.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:53.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:53.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:53.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:53.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:53.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:53.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:53.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:53.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:53.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:54.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:54.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:54.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:54.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:54.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:54.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:54.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:54.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:54.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:54.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:55.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368368 2021-05-12T12:40:55.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368368 '!=' null 2021-05-12T12:40:55.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:55.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:55.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:55.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:55.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:55.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:55.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:55.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:55.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:55.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:55.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:55.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:55.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:55.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:55.536 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:56.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:56.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:56.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:56.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:56.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:56.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:56.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:56.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:56.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:56.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989339 2021-05-12T12:40:56.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989339 '!=' null 2021-05-12T12:40:56.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:56.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:56.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:57.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:57.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:57.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:57.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:57.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:57.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:57.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:57.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:57.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:57.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:57.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:57.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:57.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:57.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:57.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:57.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:57.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:57.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:58.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368313 2021-05-12T12:40:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368313 '!=' null 2021-05-12T12:40:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:58.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:40:58.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:40:58.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:40:58.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:40:58.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:40:58.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:40:58.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:40:58.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:40:58.551 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:40:58.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:40:58.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:40:58.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:40:58.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:40:59.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:40:59.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:40:59.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:40:59.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:40:59.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:40:59.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:40:59.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:40:59.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:40:59.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:40:59.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:40:59.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:40:59.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:40:59.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:40:59.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:00.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:00.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:00.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:00.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:00.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:00.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:00.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:00.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:00.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:00.396 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:00.396 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:00.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:00.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:00.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:00.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:00.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:00.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:01.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:41:01.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:41:01.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:01.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:01.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:01.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:01.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:01.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:01.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:01.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:01.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:01.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:01.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:01.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:01.906 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:01.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:01.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:02.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:02.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:02.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:02.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:02.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:02.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:02.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:02.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:02.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:02.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115577 2021-05-12T12:41:02.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115577 '!=' null 2021-05-12T12:41:02.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:02.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:02.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:03.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:03.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:03.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:03.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:03.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:03.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:03.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:03.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:03.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:03.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:03.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:03.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:03.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:03.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:03.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:03.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:03.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:03.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:03.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:03.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:03.896 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:04.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:41:04.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:41:04.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:04.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:04.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:04.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:04.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:04.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:04.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:04.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:04.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:04.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:04.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:04.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:04.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:04.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:04.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:05.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:05.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:05.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:05.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:05.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:05.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:05.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:05.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:05.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:05.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:41:05.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:41:05.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:05.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:05.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:06.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:06.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:06.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:06.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:06.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:06.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:06.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:06.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:06.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:06.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:06.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:06.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:06.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:06.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:06.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:06.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:06.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:06.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:06.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:07.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T12:41:07.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T12:41:07.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:07.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:07.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:07.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:07.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:07.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:07.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:07.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:07.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:07.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:07.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:07.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:07.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:07.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:07.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:08.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:08.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:08.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:08.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:08.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:08.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:08.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:08.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:08.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:08.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368321 2021-05-12T12:41:08.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368321 '!=' null 2021-05-12T12:41:08.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:08.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:08.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:08.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:08.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:08.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:08.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:08.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:08.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:08.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:08.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:09.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:09.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:09.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:09.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:09.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:09.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:09.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:09.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:09.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:09.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:09.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:09.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:09.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:10.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368351 2021-05-12T12:41:10.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368351 '!=' null 2021-05-12T12:41:10.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:10.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:10.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:10.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:10.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:10.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:10.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:10.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:10.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:10.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:10.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:10.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:10.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:10.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:10.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:11.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:11.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:11.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:11.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:11.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:11.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:11.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:11.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:11.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:12.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989355 2021-05-12T12:41:12.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989355 '!=' null 2021-05-12T12:41:12.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:12.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:12.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:12.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:12.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:12.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:12.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:12.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:12.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:12.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:12.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:12.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:12.502 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:12.502 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:12.503 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:12.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:12.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:12.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:12.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:12.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:12.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:12.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:12.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:12.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:13.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989355 2021-05-12T12:41:13.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989355 '!=' null 2021-05-12T12:41:13.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:13.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:13.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:13.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:13.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:13.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:13.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:13.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:13.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:13.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:13.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:13.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:13.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:13.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:13.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:14.423 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:14.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:14.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:14.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:14.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:14.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:14.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:14.425 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:14.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:14.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242011 2021-05-12T12:41:14.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242011 '!=' null 2021-05-12T12:41:14.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:14.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:14.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:15.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:15.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:15.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:15.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:15.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:15.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:15.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:15.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:15.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:15.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:15.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:15.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:15.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:15.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:15.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:15.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:15.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:15.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:15.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:15.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:15.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:16.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862963 2021-05-12T12:41:16.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862963 '!=' null 2021-05-12T12:41:16.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:16.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:16.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:16.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:16.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:16.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:16.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:16.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:16.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:16.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:16.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:16.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:16.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:16.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:16.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:17.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:17.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:17.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:17.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:17.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:17.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:17.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:17.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:17.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241989 2021-05-12T12:41:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241989 '!=' null 2021-05-12T12:41:17.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:17.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:17.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:17.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:17.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:17.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:17.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:17.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:17.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:17.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:17.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:18.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:18.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:18.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:18.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:18.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:18.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:18.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:18.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:18.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:18.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:18.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:18.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:18.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:19.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241989 2021-05-12T12:41:19.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241989 '!=' null 2021-05-12T12:41:19.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:19.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:19.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:19.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:19.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:19.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:19.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:19.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:19.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:19.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:19.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:19.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:19.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:19.803 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:19.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:20.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:20.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:20.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:20.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:20.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:20.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:20.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:20.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:20.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241945 2021-05-12T12:41:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241945 '!=' null 2021-05-12T12:41:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:20.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:20.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:20.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:20.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:20.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:20.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:20.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:20.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:20.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:21.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:21.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:21.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:21.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:21.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:21.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:21.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:21.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862890 2021-05-12T12:41:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862890 '!=' null 2021-05-12T12:41:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:22.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:22.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:22.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:22.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:22.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:22.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:22.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:22.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:22.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:22.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:22.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:22.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:22.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:23.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:23.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:23.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:23.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:23.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:23.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:23.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:23.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:23.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:23.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862890 2021-05-12T12:41:23.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862890 '!=' null 2021-05-12T12:41:23.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:23.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:23.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:23.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:23.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:23.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:23.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:23.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:23.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:23.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:23.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:24.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:24.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:24.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:24.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:24.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:24.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:24.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:24.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:24.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:24.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:24.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:24.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:24.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:25.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241974 2021-05-12T12:41:25.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241974 '!=' null 2021-05-12T12:41:25.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:25.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:25.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:25.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:25.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:25.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:25.273 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:25.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:25.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:25.275 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:25.275 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:25.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:25.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:25.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:25.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:26.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:26.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:26.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:26.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:26.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:26.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:26.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:26.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:26.127 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:26.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862942 2021-05-12T12:41:26.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862942 '!=' null 2021-05-12T12:41:26.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:26.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:26.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:26.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:26.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:26.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:26.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:26.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:26.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:26.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:26.723 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:27.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:27.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:27.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:27.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:27.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:27.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:27.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:27.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:27.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:27.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:27.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:27.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:27.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:28.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241992 2021-05-12T12:41:28.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241992 '!=' null 2021-05-12T12:41:28.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:28.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:28.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:28.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:28.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:28.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:28.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:28.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:28.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:28.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:28.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:28.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:28.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:28.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:28.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:29.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:29.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:29.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:29.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:29.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:29.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:29.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:29.041 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:29.041 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:29.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241992 2021-05-12T12:41:29.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241992 '!=' null 2021-05-12T12:41:29.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:29.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:29.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:29.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:29.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:29.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:29.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:29.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:29.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:29.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:29.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:29.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:29.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:29.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:29.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:30.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:30.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:30.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:30.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:30.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:30.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:30.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:30.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:30.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:30.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241973 2021-05-12T12:41:30.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241973 '!=' null 2021-05-12T12:41:30.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:30.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:30.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:31.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:31.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:31.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:31.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:31.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:31.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:31.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:31.045 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:31.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:31.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:31.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:31.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:31.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:31.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:31.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:31.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:31.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:31.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:31.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:31.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:31.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:32.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989362 2021-05-12T12:41:32.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989362 '!=' null 2021-05-12T12:41:32.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:32.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:32.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:32.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:32.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:32.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:32.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:32.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:32.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:32.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:32.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:32.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:32.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:32.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:32.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:33.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:33.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:33.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:33.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:33.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:33.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:33.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:33.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T12:41:33.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T12:41:33.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:33.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:33.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:33.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:33.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:33.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:33.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:33.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:33.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:33.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:34.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:34.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:34.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:34.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:34.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:34.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:34.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:34.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:34.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:34.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:34.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:34.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:34.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:35.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T12:41:35.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T12:41:35.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:35.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:35.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:35.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:35.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:35.333 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:35.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:35.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:35.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:35.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:35.334 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:35.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:35.687 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:35.687 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:35.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:36.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:36.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:36.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:36.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:36.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989344 2021-05-12T12:41:36.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989344 '!=' null 2021-05-12T12:41:36.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:36.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:36.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:36.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:36.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:36.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:36.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:36.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:36.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:36.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:36.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:37.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:37.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:37.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:37.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:37.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:37.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:37.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:37.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:37.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:37.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:37.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:37.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:37.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:38.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T12:41:38.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T12:41:38.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:38.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:38.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:38.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:38.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:38.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:38.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:38.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:38.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:38.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:38.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:38.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:38.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:38.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:38.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:38.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:38.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:38.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:38.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:38.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:38.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:38.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:38.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:38.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:39.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T12:41:39.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T12:41:39.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:39.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:39.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:39.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:39.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:39.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:39.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:39.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:39.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:39.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:39.563 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:39.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:39.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:39.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:39.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:40.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:40.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:40.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:40.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:40.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:40.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:40.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:40.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:40.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:40.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T12:41:40.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T12:41:40.919 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:40.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:40.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:41.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:41.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:41.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:41.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:41.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:41.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:41.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:41.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:41.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:41.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:41.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:41.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:41.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:41.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:41.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:41.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:41.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:41.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:41.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:41.875 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:41.875 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:42.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989326 2021-05-12T12:41:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989326 '!=' null 2021-05-12T12:41:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:42.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:42.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:42.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:42.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:42.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:42.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:42.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:42.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:42.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:42.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:42.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:42.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:43.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:43.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:43.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:43.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:43.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:43.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:43.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:43.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:43.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:43.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241944 2021-05-12T12:41:43.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241944 '!=' null 2021-05-12T12:41:43.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:43.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:43.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:43.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:43.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:43.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:43.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:43.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:43.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:43.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:43.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:44.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:44.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:44.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:44.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:44.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:44.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:44.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:44.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:44.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:44.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:44.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:44.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:44.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:45.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241944 2021-05-12T12:41:45.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241944 '!=' null 2021-05-12T12:41:45.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:45.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:45.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:45.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:45.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:45.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:45.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:45.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:45.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:45.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:45.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:45.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:45.866 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:45.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:45.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:46.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:46.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:46.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:46.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:46.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:46.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989303 2021-05-12T12:41:46.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989303 '!=' null 2021-05-12T12:41:46.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:46.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:46.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:46.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:47.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:47.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:47.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:47.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:47.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:47.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:47.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:47.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:47.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:47.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:47.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:47.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:47.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:47.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:47.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:47.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:47.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:47.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:47.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:47.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:48.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368290 2021-05-12T12:41:48.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368290 '!=' null 2021-05-12T12:41:48.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:48.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:48.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:48.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:48.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:48.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:48.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:48.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:48.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:48.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:48.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:48.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:48.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:48.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:48.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:49.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:49.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:49.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:49.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:49.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:49.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:49.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:49.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:49.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:50.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T12:41:50.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T12:41:50.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:50.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:50.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:50.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:50.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:50.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:50.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:50.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:50.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:50.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:50.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:50.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:50.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:50.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:50.510 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:50.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:50.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:50.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:50.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:50.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:50.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:50.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:50.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:50.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:51.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T12:41:51.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T12:41:51.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:51.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:51.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:51.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:51.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:51.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:51.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:51.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:51.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:51.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:51.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:51.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:51.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:51.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:51.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:52.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:52.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:52.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:52.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:52.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:52.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:52.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:52.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:52.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:52.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989281 2021-05-12T12:41:52.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989281 '!=' null 2021-05-12T12:41:52.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:52.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:52.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:52.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:52.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:52.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:52.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:52.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:52.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:52.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:52.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:53.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:53.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:53.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:53.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:53.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:53.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:53.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:53.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:53.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:53.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:53.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:53.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:53.826 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368273 2021-05-12T12:41:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368273 '!=' null 2021-05-12T12:41:54.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:54.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:54.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:54.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:54.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:54.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:54.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:54.462 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:54.462 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:54.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:54.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:54.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:54.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:55.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:55.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:55.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:55.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:55.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:55.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:55.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:55.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:55.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:55.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862954 2021-05-12T12:41:55.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862954 '!=' null 2021-05-12T12:41:55.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:55.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:55.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:55.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:55.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:55.955 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:55.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:55.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:55.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:55.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:55.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:56.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:56.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:56.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:56.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:56.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:56.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:56.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:56.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:56.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:56.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:56.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:56.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:56.806 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:57.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862954 2021-05-12T12:41:57.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862954 '!=' null 2021-05-12T12:41:57.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:57.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:57.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:57.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:57.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:57.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:57.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:57.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:57.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:57.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:57.390 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:57.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:57.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:57.754 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:57.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:58.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:58.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:58.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:58.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:58.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:58.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:58.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:58.247 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:58.247 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:41:58.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115564 2021-05-12T12:41:58.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115564 '!=' null 2021-05-12T12:41:58.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:41:58.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:41:58.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:41:58.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:41:58.835 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:41:58.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:41:58.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:41:58.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:41:58.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:41:58.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:41:58.838 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:41:59.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:41:59.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:41:59.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:41:59.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:41:59.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:41:59.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:41:59.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:41:59.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:41:59.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:41:59.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:41:59.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:41:59.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:41:59.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:00.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115587 2021-05-12T12:42:00.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115587 '!=' null 2021-05-12T12:42:00.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:00.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:00.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:00.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:00.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:00.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:00.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:00.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:00.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:00.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:00.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:00.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:00.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:00.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:00.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:01.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:01.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:01.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:01.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:01.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:01.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:01.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:01.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:01.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:01.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115587 2021-05-12T12:42:01.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115587 '!=' null 2021-05-12T12:42:01.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:01.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:01.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:01.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:01.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:01.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:01.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:01.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:01.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:01.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:01.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:02.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:02.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:02.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:02.082 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:02.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:02.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:02.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:02.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:02.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:02.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:02.560 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:02.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:02.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:03.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736521 2021-05-12T12:42:03.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736521 '!=' null 2021-05-12T12:42:03.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:03.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:03.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:03.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:03.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:03.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:03.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:03.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:03.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:03.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:03.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:03.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:03.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:03.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:03.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:03.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:03.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:03.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:03.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:03.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:03.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:03.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:03.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:03.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:04.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115565 2021-05-12T12:42:04.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115565 '!=' null 2021-05-12T12:42:04.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:04.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:04.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:04.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:04.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:04.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:04.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:04.585 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:04.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:04.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:04.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:04.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:04.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:04.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:04.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:05.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:05.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:05.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:05.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:05.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:05.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:05.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:05.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:05.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:05.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862930 2021-05-12T12:42:05.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862930 '!=' null 2021-05-12T12:42:05.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:05.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:05.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:06.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:06.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:06.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:06.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:06.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:06.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:06.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:06.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:06.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:06.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:06.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:06.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:06.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:06.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:06.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:06.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:06.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:06.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:06.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:06.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:06.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:07.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862930 2021-05-12T12:42:07.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862930 '!=' null 2021-05-12T12:42:07.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:07.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:07.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:07.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:07.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:07.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:07.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:07.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:07.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:07.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:07.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:07.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:07.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:07.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:07.833 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:08.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:08.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:08.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:08.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:08.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:08.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:08.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:08.326 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:08.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:08.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368276 2021-05-12T12:42:08.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368276 '!=' null 2021-05-12T12:42:08.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:08.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:08.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:08.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:08.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:08.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:09.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:09.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:09.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:09.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:09.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:09.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:09.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:09.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:09.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:09.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:09.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:09.769 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:09.770 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:10.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:10.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:10.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:10.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:10.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:10.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:10.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:10.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:10.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:10.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:10.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:10.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:10.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:10.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:10.710 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:11.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:11.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:11.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:11.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:11.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:11.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:11.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:11.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:11.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:11.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:42:11.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:42:11.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:11.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:11.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:12.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:12.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:12.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:12.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:12.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:12.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:12.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:12.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:12.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:12.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:12.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:12.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:12.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:12.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:12.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:12.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:12.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:12.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:12.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:12.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:12.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:13.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989336 2021-05-12T12:42:13.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989336 '!=' null 2021-05-12T12:42:13.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:13.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:13.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:13.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:13.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:13.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:13.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:13.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:13.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:13.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:13.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:13.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:13.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:13.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:13.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:14.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:14.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:14.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:14.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:14.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:14.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:14.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:14.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:14.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:14.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:42:14.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:42:14.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:14.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:14.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:15.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:15.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:15.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:15.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:15.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:15.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:15.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:15.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:15.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:15.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:15.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:15.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:15.828 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:15.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:15.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:15.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:15.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:15.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:15.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:15.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:15.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:16.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115638 2021-05-12T12:42:16.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115638 '!=' null 2021-05-12T12:42:16.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:16.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:16.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:16.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:16.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:16.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:16.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:16.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:16.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:16.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:16.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:16.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:16.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:16.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:17.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:17.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:17.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:17.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:17.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:17.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:17.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:17.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:17.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:17.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115638 2021-05-12T12:42:17.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115638 '!=' null 2021-05-12T12:42:17.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:17.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:17.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:17.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:17.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:17.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:17.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:17.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:17.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:17.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:17.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:18.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:18.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:18.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:18.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:18.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:18.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:18.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:18.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:18.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:18.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:18.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:18.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:18.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:19.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:42:19.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:42:19.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:19.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:19.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:19.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:19.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:19.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:19.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:19.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:19.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:19.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:19.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:19.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:19.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:19.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:19.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:20.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:20.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:20.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:20.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:20.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:20.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:20.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:20.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:20.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:20.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T12:42:20.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T12:42:20.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:20.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:20.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:20.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:20.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:20.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:20.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:20.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:20.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:20.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:20.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:21.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:21.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:21.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:21.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:21.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:21.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:21.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:21.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:21.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:21.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:21.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:21.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:21.576 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:22.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989203 2021-05-12T12:42:22.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989203 '!=' null 2021-05-12T12:42:22.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:22.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:22.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:22.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:22.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:22.250 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:22.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:22.251 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:22.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:22.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:22.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:22.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:22.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:22.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:22.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:23.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:23.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:23.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:23.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:23.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:23.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:23.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:23.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:23.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:23.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989203 2021-05-12T12:42:23.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989203 '!=' null 2021-05-12T12:42:23.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:23.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:23.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:23.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:23.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:23.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:23.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:23.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:23.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:23.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:23.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:24.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:24.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:24.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:24.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:24.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:24.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:24.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:24.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:24.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:24.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:24.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:24.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:24.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:25.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368217 2021-05-12T12:42:25.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368217 '!=' null 2021-05-12T12:42:25.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:25.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:25.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:25.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:25.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:25.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:25.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:25.222 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:25.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:25.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:25.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:25.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:25.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:25.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:25.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:26.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:26.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:26.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:26.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:26.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:26.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989185 2021-05-12T12:42:26.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989185 '!=' null 2021-05-12T12:42:26.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:26.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:26.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:26.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:26.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:26.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:26.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:26.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:26.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:26.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:26.650 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:27.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:27.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:27.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:27.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:27.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:27.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:27.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:27.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:27.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:27.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:27.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:27.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:27.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:28.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T12:42:28.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T12:42:28.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:28.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:28.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:28.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:28.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:28.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:28.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:28.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:28.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:28.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:28.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:28.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:28.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:28.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:28.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:29.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:29.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:29.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:29.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:29.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:29.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:29.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:29.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:29.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:29.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T12:42:29.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T12:42:29.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:29.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:29.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:29.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:29.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:29.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:29.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:29.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:29.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:29.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:29.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:29.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:29.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:29.994 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:29.995 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:30.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:30.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:30.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:30.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:30.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:30.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:30.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:30.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:30.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241939 2021-05-12T12:42:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241939 '!=' null 2021-05-12T12:42:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:30.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:31.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:31.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:31.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:31.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:31.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:31.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:31.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:31.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:31.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:31.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:31.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:31.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:31.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:31.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:31.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:31.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:31.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:31.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:31.937 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:32.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862939 2021-05-12T12:42:32.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862939 '!=' null 2021-05-12T12:42:32.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:32.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:32.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:32.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:32.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:32.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:32.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:32.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:32.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:32.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:32.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:32.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:32.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:32.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:32.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:33.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:33.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:33.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:33.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:33.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:33.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241987 2021-05-12T12:42:33.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241987 '!=' null 2021-05-12T12:42:33.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:33.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:33.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:33.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:33.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:33.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:33.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:33.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:33.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:33.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:33.984 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:34.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:34.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:34.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:34.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:34.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:34.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:34.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:34.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:34.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:34.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:34.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:34.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:34.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:35.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241987 2021-05-12T12:42:35.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241987 '!=' null 2021-05-12T12:42:35.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:35.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:35.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:35.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:35.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:35.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:35.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:35.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:35.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:35.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:35.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:35.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:35.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:35.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:36.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:36.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:36.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:36.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:36.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:36.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:36.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:36.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863031 2021-05-12T12:42:36.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863031 '!=' null 2021-05-12T12:42:36.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:36.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:36.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:37.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:37.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:37.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:37.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:37.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:37.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:37.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:37.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:37.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:37.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:37.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:37.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:37.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:37.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:37.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:37.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:37.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:37.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:37.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:38.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242007 2021-05-12T12:42:38.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242007 '!=' null 2021-05-12T12:42:38.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:38.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:38.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:38.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:38.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:38.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:38.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:38.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:38.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:38.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:38.462 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:38.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:38.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:38.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:38.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:39.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:39.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:39.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:39.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:39.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:39.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:39.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:39.340 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:39.341 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:39.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242007 2021-05-12T12:42:39.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242007 '!=' null 2021-05-12T12:42:39.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:39.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:39.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:39.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:39.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:39.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:39.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:39.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:39.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:39.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:39.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:40.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:40.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:40.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:40.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:40.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:40.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:40.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:40.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:40.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:40.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:40.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:40.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:40.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:41.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242016 2021-05-12T12:42:41.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242016 '!=' null 2021-05-12T12:42:41.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:41.253 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:41.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:41.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:41.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:41.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:41.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:41.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:41.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:41.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:41.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:41.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:41.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:41.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:41.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:42.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:42.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:42.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:42.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:42.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:42.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:42.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:42.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:42.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863062 2021-05-12T12:42:42.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863062 '!=' null 2021-05-12T12:42:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:42.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:42.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:42.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:42.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:42.786 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:42.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:42.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:42.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:42.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:43.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:43.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:43.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:43.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:43.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:43.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:43.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:43.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:43.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:43.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:43.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:43.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:43.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:44.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242050 2021-05-12T12:42:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242050 '!=' null 2021-05-12T12:42:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:44.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:44.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:44.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:44.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:44.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:44.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:44.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:44.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:44.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:44.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:44.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:44.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:44.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:45.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:45.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:45.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:45.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:45.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:45.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:45.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:45.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:45.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:45.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242050 2021-05-12T12:42:45.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242050 '!=' null 2021-05-12T12:42:45.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:45.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:45.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:45.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:45.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:45.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:45.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:45.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:45.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:45.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:45.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:46.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:46.090 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:46.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:46.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:46.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:46.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:46.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:46.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:46.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:46.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:46.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:46.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:46.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:47.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989404 2021-05-12T12:42:47.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989404 '!=' null 2021-05-12T12:42:47.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:47.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:47.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:47.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:47.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:47.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:47.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:47.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:47.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:47.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:47.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:47.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:47.537 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:47.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:47.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:48.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:48.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:48.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:48.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:48.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:48.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:48.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:48.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:48.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:48.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368367 2021-05-12T12:42:48.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368367 '!=' null 2021-05-12T12:42:48.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:48.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:48.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:48.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:48.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:48.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:48.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:48.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:48.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:48.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:48.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:48.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:48.955 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:48.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:48.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:49.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:49.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:49.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:49.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:49.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:49.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:49.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:49.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:49.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:49.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T12:42:49.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T12:42:49.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:49.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:49.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:50.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:50.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:50.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:50.022 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:50.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:50.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:50.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:50.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:50.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:50.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:50.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:50.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:50.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:50.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:50.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:50.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:50.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:50.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:50.875 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:50.876 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:51.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T12:42:51.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T12:42:51.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:51.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:51.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:51.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:51.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:51.453 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:51.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:51.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:51.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:51.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:51.454 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:51.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:51.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:51.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:51.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:52.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:52.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:52.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:52.332 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:52.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:52.333 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:52.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T12:42:52.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T12:42:52.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:52.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:52.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:52.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:52.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:52.926 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:52.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:52.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:52.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:52.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:52.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:53.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:53.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:53.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:53.291 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:53.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:53.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:53.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:53.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:53.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:53.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:53.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:53.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:53.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:54.248 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T12:42:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T12:42:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:54.249 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:54.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:54.350 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:54.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:54.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:54.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:54.351 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:54.352 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:54.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:54.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:54.716 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:54.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:55.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:55.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:55.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:55.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:55.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:55.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:55.196 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:55.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:55.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:55.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T12:42:55.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T12:42:55.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:55.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:55.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:55.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:55.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:55.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:55.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:55.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:55.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:55.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:55.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:56.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:56.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:56.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:56.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:56.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:56.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:56.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:56.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:56.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:56.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:56.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:56.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:56.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:57.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989261 2021-05-12T12:42:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989261 '!=' null 2021-05-12T12:42:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:57.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:57.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:57.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:57.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:57.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:57.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:57.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:57.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:57.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:57.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:57.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:57.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:57.589 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:58.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:58.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:58.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:58.062 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:58.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:58.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:58.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:58.063 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:58.064 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:42:58.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241949 2021-05-12T12:42:58.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241949 '!=' null 2021-05-12T12:42:58.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:42:58.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:42:58.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:42:58.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:42:58.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:42:58.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:42:58.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:42:58.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:42:58.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:42:58.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:42:58.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:42:58.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:42:58.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:42:58.997 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:42:58.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:42:59.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:42:59.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:42:59.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:42:59.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:42:59.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:42:59.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:42:59.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:42:59.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:42:59.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:00.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241960 2021-05-12T12:43:00.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241960 '!=' null 2021-05-12T12:43:00.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:00.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:00.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:00.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:00.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:00.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:00.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:00.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:00.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:00.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:00.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:00.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:00.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:00.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:01.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:01.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:01.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:01.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:01.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:01.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:01.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:01.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:01.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:01.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241960 2021-05-12T12:43:01.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241960 '!=' null 2021-05-12T12:43:01.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:01.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:01.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:01.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:01.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:01.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:01.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:01.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:01.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:01.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:01.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:01.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:01.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:01.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:01.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:02.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:02.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:02.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:02.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:02.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:02.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:02.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:02.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:02.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:02.954 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989279 2021-05-12T12:43:02.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989279 '!=' null 2021-05-12T12:43:02.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:02.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:02.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:03.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:03.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:03.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:03.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:03.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:03.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:03.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:03.058 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:03.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:03.426 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:03.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:03.427 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:04.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:04.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:04.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:04.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:04.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368331 2021-05-12T12:43:04.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368331 '!=' null 2021-05-12T12:43:04.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:04.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:04.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:04.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:04.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:04.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:04.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:04.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:04.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:04.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:04.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:04.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:04.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:04.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:04.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:05.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:05.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:05.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:05.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:05.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:05.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:05.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:05.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:05.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:05.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989340 2021-05-12T12:43:05.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989340 '!=' null 2021-05-12T12:43:05.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:05.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:05.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:06.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:06.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:06.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:06.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:06.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:06.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:06.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:06.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:06.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:06.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:06.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:06.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:06.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:06.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:06.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:06.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:06.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:06.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:06.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:06.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:06.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:07.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989340 2021-05-12T12:43:07.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989340 '!=' null 2021-05-12T12:43:07.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:07.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:07.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:07.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:07.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:07.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:07.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:07.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:07.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:07.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:07.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:07.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:07.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:07.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:07.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:08.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:08.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:08.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:08.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:08.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:08.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:08.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:08.330 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:08.330 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:08.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368328 2021-05-12T12:43:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368328 '!=' null 2021-05-12T12:43:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:08.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:08.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:08.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:08.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:08.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:08.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:08.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:08.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:08.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:09.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:09.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:09.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:09.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:09.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:09.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:09.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:09.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:09.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:09.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:09.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:09.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:09.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:10.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368336 2021-05-12T12:43:10.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368336 '!=' null 2021-05-12T12:43:10.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:10.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:10.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:10.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:10.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:10.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:10.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:10.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:10.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:10.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:10.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:10.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:10.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:10.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:10.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:11.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:11.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:11.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:11.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:11.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:11.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:11.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:11.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:11.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:11.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862971 2021-05-12T12:43:11.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862971 '!=' null 2021-05-12T12:43:11.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:11.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:11.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:11.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:11.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:11.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:11.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:11.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:12.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:12.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:12.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:12.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:12.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:12.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:12.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:12.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:12.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:12.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:12.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:12.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:12.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:13.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862971 2021-05-12T12:43:13.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862971 '!=' null 2021-05-12T12:43:13.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:13.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:13.584 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:13.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:13.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:13.687 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:13.687 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:13.687 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:13.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:13.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:13.688 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:14.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:14.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:14.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:14.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:14.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:14.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:14.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:14.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:14.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:14.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:14.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:14.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:14.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:15.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115589 2021-05-12T12:43:15.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115589 '!=' null 2021-05-12T12:43:15.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:15.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:15.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:15.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:15.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:15.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:15.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:15.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:15.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:15.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:15.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:15.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:15.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:15.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:15.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:16.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:16.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:16.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:16.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:16.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:16.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:16.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:16.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:16.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:16.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736574 2021-05-12T12:43:16.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736574 '!=' null 2021-05-12T12:43:16.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:16.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:16.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:16.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:16.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:16.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:16.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:16.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:16.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:16.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:16.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:16.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:16.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:16.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:16.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:17.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:17.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:17.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:17.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:17.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:17.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:17.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:17.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:17.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:17.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115600 2021-05-12T12:43:17.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115600 '!=' null 2021-05-12T12:43:17.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:17.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:17.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:18.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:18.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:18.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:18.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:18.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:18.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:18.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:18.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:18.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:18.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:18.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:18.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:18.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:18.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:18.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:18.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:18.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:18.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:18.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:18.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:18.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:19.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115600 2021-05-12T12:43:19.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115600 '!=' null 2021-05-12T12:43:19.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:19.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:19.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:19.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:19.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:19.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:19.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:19.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:19.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:19.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:19.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:19.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:19.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:19.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:20.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:20.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:20.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:20.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:20.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:20.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:20.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:20.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:20.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115608 2021-05-12T12:43:20.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115608 '!=' null 2021-05-12T12:43:20.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:20.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:20.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:20.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:20.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:20.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:20.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:20.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:20.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:20.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:20.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:21.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:21.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:21.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:21.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:21.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:21.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:21.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:21.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:21.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:21.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:21.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:21.862 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:21.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:22.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862931 2021-05-12T12:43:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862931 '!=' null 2021-05-12T12:43:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:22.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:22.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:22.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:22.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:22.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:22.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:22.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:22.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:22.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:22.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:22.831 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:22.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:22.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:23.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:23.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:23.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:23.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:23.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:23.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:24.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368294 2021-05-12T12:43:24.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368294 '!=' null 2021-05-12T12:43:24.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:24.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:24.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:24.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:24.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:24.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:24.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:24.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:24.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:24.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:24.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:24.478 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:24.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:24.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:24.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:24.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:24.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:24.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:24.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:24.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:24.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:25.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368294 2021-05-12T12:43:25.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368294 '!=' null 2021-05-12T12:43:25.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:25.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:25.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:25.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:25.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:25.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:25.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:25.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:25.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:25.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:25.545 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:25.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:25.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:25.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:25.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:26.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:26.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:26.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:26.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:26.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:26.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:26.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:26.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:26.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:26.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989315 2021-05-12T12:43:26.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989315 '!=' null 2021-05-12T12:43:26.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:26.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:26.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:26.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:26.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:26.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:26.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:26.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:26.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:26.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:26.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:27.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:27.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:27.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:27.346 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:27.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:27.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:27.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:27.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:27.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:27.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:27.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:27.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:27.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:28.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368319 2021-05-12T12:43:28.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368319 '!=' null 2021-05-12T12:43:28.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:28.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:28.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:28.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:28.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:28.490 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:28.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:28.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:28.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:28.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:28.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:28.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:28.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:28.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:28.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:29.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:29.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:29.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:29.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:29.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:29.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:29.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368356 2021-05-12T12:43:29.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368356 '!=' null 2021-05-12T12:43:29.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:29.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:29.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:29.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:29.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:29.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:29.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:29.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:29.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:29.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:29.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:30.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:30.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:30.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:30.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:30.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:30.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:30.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:30.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:30.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:30.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:30.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:30.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:30.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:31.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368356 2021-05-12T12:43:31.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368356 '!=' null 2021-05-12T12:43:31.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:31.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:31.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:31.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:31.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:31.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:31.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:31.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:31.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:31.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:31.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:31.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:31.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:31.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:31.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:32.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:32.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:32.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:32.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:32.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:32.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:32.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:32.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:32.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115715 2021-05-12T12:43:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115715 '!=' null 2021-05-12T12:43:33.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:33.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:33.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:33.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:33.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:33.249 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:33.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:33.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:33.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:33.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:34.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:34.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:34.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:34.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:34.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:34.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:34.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:34.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:34.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368363 2021-05-12T12:43:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368363 '!=' null 2021-05-12T12:43:34.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:34.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:34.575 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:34.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:34.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:34.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:34.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:34.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:34.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:34.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:34.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:35.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:35.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:35.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:35.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:35.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:35.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:35.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:35.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:35.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:35.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:35.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989407 2021-05-12T12:43:35.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989407 '!=' null 2021-05-12T12:43:35.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:35.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:35.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:36.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:36.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:36.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:36.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:36.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:36.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:36.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:36.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:36.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:36.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:36.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:36.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:36.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:36.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:36.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:36.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:36.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:36.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:37.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989407 2021-05-12T12:43:37.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989407 '!=' null 2021-05-12T12:43:37.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:37.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:37.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:37.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:37.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:37.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:37.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:37.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:37.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:37.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:37.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:37.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:37.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:37.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:37.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:38.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:38.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:38.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:38.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368380 2021-05-12T12:43:38.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368380 '!=' null 2021-05-12T12:43:38.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:38.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:38.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:39.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:39.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:39.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:39.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:39.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:39.018 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:39.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:39.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:39.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:39.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:39.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:39.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:39.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:39.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:39.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:39.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:39.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:39.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:39.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:39.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:39.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:40.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368382 2021-05-12T12:43:40.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368382 '!=' null 2021-05-12T12:43:40.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:40.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:40.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:40.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:40.542 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:40.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:40.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:40.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:40.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:40.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:40.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:40.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:40.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:40.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:40.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:41.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:41.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:41.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:41.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:41.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:41.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:41.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:41.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:41.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989352 2021-05-12T12:43:41.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989352 '!=' null 2021-05-12T12:43:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:41.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:42.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:42.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:42.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:42.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:42.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:42.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:42.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:42.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:42.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:42.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:42.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:42.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:42.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:42.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:42.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:42.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:42.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:42.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:42.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:42.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:42.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:43.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989352 2021-05-12T12:43:43.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989352 '!=' null 2021-05-12T12:43:43.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:43.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:43.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:43.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:43.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:43.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:43.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:43.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:43.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:43.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:43.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:43.901 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:43.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:43.902 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:43.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:44.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:44.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:44.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:44.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:44.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:44.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:44.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241994 2021-05-12T12:43:44.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241994 '!=' null 2021-05-12T12:43:44.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:44.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:44.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:44.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:44.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:44.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:44.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:44.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:44.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:44.970 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:44.970 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:45.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:45.326 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:45.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:45.327 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:45.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:45.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:45.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:45.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:45.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:45.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:45.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:45.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:45.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:46.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862911 2021-05-12T12:43:46.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862911 '!=' null 2021-05-12T12:43:46.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:46.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:46.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:46.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:46.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:46.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:46.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:46.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:46.470 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:46.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:46.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:46.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:46.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:46.824 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:47.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:47.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:47.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:47.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:47.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:47.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:47.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:47.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:47.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:47.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862911 2021-05-12T12:43:47.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862911 '!=' null 2021-05-12T12:43:47.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:47.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:47.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:47.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:47.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:47.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:47.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:47.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:47.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:47.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:47.886 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:48.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:48.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:48.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:48.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:48.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:48.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:48.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:48.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:48.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:48.724 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:48.724 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:49.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241939 2021-05-12T12:43:49.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241939 '!=' null 2021-05-12T12:43:49.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:49.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:49.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:49.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:49.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:49.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:49.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:49.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:49.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:49.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:49.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:49.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:49.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:49.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:49.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:50.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:50.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:50.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:50.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:50.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241952 2021-05-12T12:43:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241952 '!=' null 2021-05-12T12:43:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:50.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:50.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:50.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:50.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:50.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:50.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:50.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:50.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:50.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:51.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:51.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:51.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:51.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:51.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:51.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:51.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:51.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:51.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:51.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:52.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862907 2021-05-12T12:43:52.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862907 '!=' null 2021-05-12T12:43:52.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:52.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:52.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:52.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:52.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:52.160 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:52.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:52.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:52.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:52.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:52.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:52.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:52.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:52.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:52.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:53.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:53.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:53.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:53.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:53.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:53.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:53.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:53.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:53.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:53.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862907 2021-05-12T12:43:53.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862907 '!=' null 2021-05-12T12:43:53.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:53.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:53.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:53.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:53.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:53.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:53.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:53.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:53.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:53.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:53.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:53.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:53.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:53.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:53.948 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:54.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:54.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:54.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:54.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:54.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241957 2021-05-12T12:43:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241957 '!=' null 2021-05-12T12:43:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:54.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:54.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:55.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:55.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:55.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:55.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:55.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:55.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:55.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:55.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:55.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:55.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:55.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:55.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:55.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:55.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:55.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:55.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:55.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:55.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:55.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:55.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:55.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:56.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862948 2021-05-12T12:43:56.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862948 '!=' null 2021-05-12T12:43:56.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:56.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:56.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:56.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:56.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:56.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:56.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:56.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:56.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:56.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:56.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:56.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:56.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:56.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:56.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:57.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:57.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:57.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:57.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:57.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:57.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:57.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:57.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:57.272 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:57.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862948 2021-05-12T12:43:57.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862948 '!=' null 2021-05-12T12:43:57.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:57.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:57.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:57.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:57.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:57.858 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:57.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:57.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:57.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:57.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:57.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:58.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:58.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:58.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:58.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:43:58.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:43:58.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:43:58.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:43:58.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:43:58.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:43:58.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:43:58.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:43:58.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:43:58.777 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:43:59.271 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241995 2021-05-12T12:43:59.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241995 '!=' null 2021-05-12T12:43:59.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:43:59.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:43:59.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:43:59.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:43:59.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:43:59.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:43:59.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:43:59.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:43:59.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:43:59.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:43:59.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:43:59.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:43:59.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:43:59.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:43:59.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:00.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:00.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:00.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:00.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:00.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:00.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:00.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:00.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241999 2021-05-12T12:44:00.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241999 '!=' null 2021-05-12T12:44:00.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:00.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:00.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:00.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:00.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:00.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:00.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:00.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:00.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:00.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:00.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:01.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:01.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:01.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:01.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:01.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:01.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:01.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:01.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:01.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:01.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:01.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:01.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:01.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:02.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862995 2021-05-12T12:44:02.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862995 '!=' null 2021-05-12T12:44:02.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:02.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:02.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:02.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:02.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:02.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:02.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:02.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:02.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:02.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:03.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:03.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:03.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:03.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:03.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:03.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:03.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:03.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:03.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:03.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862995 2021-05-12T12:44:03.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862995 '!=' null 2021-05-12T12:44:03.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:03.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:03.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:03.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:03.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:03.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:03.674 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:03.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:03.675 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:04.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:04.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:04.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:04.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:04.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:04.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:04.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:04.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:04.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:04.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:04.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:04.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:04.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:05.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242013 2021-05-12T12:44:05.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242013 '!=' null 2021-05-12T12:44:05.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:05.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:05.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:05.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:05.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:05.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:05.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:05.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:05.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:05.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:05.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:05.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:05.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:05.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:05.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:06.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:06.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:06.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:06.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:06.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:06.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:06.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:06.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:06.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989374 2021-05-12T12:44:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989374 '!=' null 2021-05-12T12:44:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:06.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:06.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:06.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:06.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:06.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:06.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:06.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:06.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:06.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:06.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:06.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:06.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:06.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:06.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:07.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:07.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:07.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:07.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:07.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:07.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:07.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:07.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:07.428 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:07.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368352 2021-05-12T12:44:07.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368352 '!=' null 2021-05-12T12:44:07.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:07.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:07.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:08.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:08.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:08.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:08.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:08.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:08.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:08.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:08.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:08.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:08.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:08.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:08.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:08.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:08.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:08.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:08.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:08.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:08.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:08.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:08.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:08.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:09.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368352 2021-05-12T12:44:09.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368352 '!=' null 2021-05-12T12:44:09.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:09.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:09.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:09.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:09.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:09.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:09.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:09.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:09.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:09.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:09.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:09.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:09.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:09.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:09.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:10.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:10.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:10.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:10.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:10.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:10.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:10.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:10.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:10.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:10.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368364 2021-05-12T12:44:10.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368364 '!=' null 2021-05-12T12:44:10.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:10.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:10.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:11.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:11.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:11.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:11.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:11.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:11.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:11.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:11.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:11.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:11.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:11.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:11.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:11.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:11.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:11.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:11.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:11.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:11.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:11.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:11.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:11.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:12.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989375 2021-05-12T12:44:12.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989375 '!=' null 2021-05-12T12:44:12.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:12.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:12.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:12.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:12.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:12.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:12.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:12.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:12.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:12.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:12.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:12.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:12.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:12.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:12.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:13.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:13.329 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:13.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:13.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:13.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:13.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:13.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:13.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:13.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:13.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:44:13.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:44:13.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:13.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:13.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:13.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:13.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:13.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:13.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:13.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:13.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:13.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:13.920 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:14.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:14.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:14.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:14.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:14.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:14.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:14.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:14.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:14.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:14.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:14.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:14.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:14.927 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:15.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:44:15.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:44:15.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:15.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:15.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:15.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:15.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:15.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:15.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:15.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:15.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:15.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:15.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:15.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:15.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:15.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:15.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:16.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:16.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:16.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:16.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:16.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:16.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:16.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:16.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:16.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:16.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989316 2021-05-12T12:44:16.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989316 '!=' null 2021-05-12T12:44:16.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:16.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:16.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:16.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:16.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:16.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:16.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:16.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:16.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:16.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:16.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:17.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:17.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:17.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:17.349 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:17.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:17.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:17.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:17.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:17.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:17.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:17.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:17.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:17.833 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:18.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241988 2021-05-12T12:44:18.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241988 '!=' null 2021-05-12T12:44:18.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:18.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:18.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:18.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:18.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:18.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:18.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:18.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:18.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:18.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:18.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:18.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:18.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:18.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:18.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:19.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:19.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:19.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:19.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:19.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:19.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:19.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:19.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:19.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:19.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241988 2021-05-12T12:44:19.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241988 '!=' null 2021-05-12T12:44:19.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:19.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:19.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:19.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:19.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:19.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:19.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:19.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:19.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:19.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:19.879 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:20.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:20.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:20.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:20.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:20.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:20.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:20.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:20.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:20.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:20.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:20.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:20.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:20.849 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:21.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241995 2021-05-12T12:44:21.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241995 '!=' null 2021-05-12T12:44:21.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:21.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:21.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:21.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:21.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:21.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:21.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:21.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:21.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:21.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:21.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:21.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:21.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:21.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:21.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:22.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:22.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:22.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:22.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:22.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:22.289 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:22.289 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:22.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989259 2021-05-12T12:44:22.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989259 '!=' null 2021-05-12T12:44:22.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:22.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:22.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:22.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:22.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:22.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:22.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:22.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:22.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:22.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:22.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:23.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:23.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:23.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:23.230 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:23.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:23.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:23.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:23.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:23.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:23.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:23.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:23.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:23.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:24.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368291 2021-05-12T12:44:24.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368291 '!=' null 2021-05-12T12:44:24.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:24.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:24.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:24.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:24.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:24.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:24.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:24.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:24.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:24.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:24.348 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:24.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:24.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:24.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:24.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:25.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:25.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:25.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:25.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:25.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:25.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:25.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:25.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:25.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:25.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368291 2021-05-12T12:44:25.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368291 '!=' null 2021-05-12T12:44:25.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:25.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:25.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:25.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:25.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:25.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:25.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:25.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:25.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:25.763 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:25.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:26.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:26.123 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:26.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:26.124 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:26.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:26.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:26.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:26.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:26.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:26.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:26.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:26.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:26.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:27.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989295 2021-05-12T12:44:27.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989295 '!=' null 2021-05-12T12:44:27.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:27.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:27.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:27.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:27.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:27.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:27.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:27.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:27.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:27.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:27.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:27.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:27.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:27.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:27.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:28.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:28.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:28.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:28.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:28.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:28.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:28.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:28.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:28.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:28.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T12:44:28.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T12:44:28.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:28.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:28.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:28.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:28.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:28.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:28.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:28.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:28.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:28.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:28.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:29.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:29.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:29.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:29.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:29.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:29.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:29.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:29.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:29.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:29.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:29.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:29.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:29.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:30.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368296 2021-05-12T12:44:30.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368296 '!=' null 2021-05-12T12:44:30.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:30.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:30.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:30.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:30.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:30.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:30.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:30.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:30.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:30.152 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:30.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:30.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:30.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:30.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:30.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:30.990 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:30.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:30.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:30.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:30.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:30.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:31.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368296 2021-05-12T12:44:31.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368296 '!=' null 2021-05-12T12:44:31.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:31.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:31.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:31.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:31.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:31.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:31.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:31.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:31.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:31.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:31.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:32.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:32.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:32.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:32.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:32.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:32.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:32.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:32.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:32.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:32.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:32.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:32.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:32.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:33.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862906 2021-05-12T12:44:33.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862906 '!=' null 2021-05-12T12:44:33.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:33.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:33.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:33.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:33.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:33.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:33.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:33.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:33.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:33.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:33.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:33.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:33.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:33.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:33.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:34.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:34.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:34.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:34.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:34.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:34.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:34.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:34.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:34.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:34.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115641 2021-05-12T12:44:34.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115641 '!=' null 2021-05-12T12:44:34.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:34.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:34.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:34.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:34.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:34.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:34.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:34.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:34.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:34.836 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:34.837 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:35.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:35.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:35.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:35.197 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:35.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:35.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:35.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:35.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:35.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:35.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:35.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:35.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:35.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:36.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736625 2021-05-12T12:44:36.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736625 '!=' null 2021-05-12T12:44:36.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:36.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:36.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:36.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:36.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:36.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:36.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:36.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:36.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:36.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:36.326 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:36.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:36.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:36.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:36.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:37.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:37.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:37.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:37.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:37.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:37.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:37.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:37.190 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:37.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736625 2021-05-12T12:44:37.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736625 '!=' null 2021-05-12T12:44:37.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:37.673 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:37.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:37.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:37.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:37.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:37.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:37.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:37.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:37.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:38.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:38.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:38.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:38.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:38.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:38.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:38.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:38.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115653 2021-05-12T12:44:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115653 '!=' null 2021-05-12T12:44:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:39.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:39.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:39.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:39.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:39.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:39.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:39.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:39.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:39.215 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:39.215 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:39.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:39.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:39.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:39.586 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:40.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:40.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:40.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:40.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:40.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:40.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:40.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:40.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:40.078 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115664 2021-05-12T12:44:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115664 '!=' null 2021-05-12T12:44:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:40.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:40.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:40.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:40.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:40.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:40.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:40.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:40.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:40.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:40.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:41.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:41.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:41.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:41.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:41.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:41.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:41.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:41.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:41.595 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:41.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:42.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863050 2021-05-12T12:44:42.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863050 '!=' null 2021-05-12T12:44:42.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:42.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:42.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:42.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:42.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:42.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:42.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:42.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:42.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:42.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:42.194 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:42.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:42.550 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:42.551 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:42.551 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:43.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:43.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:43.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:43.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:43.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:43.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:43.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:43.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:43.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:43.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863050 2021-05-12T12:44:43.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863050 '!=' null 2021-05-12T12:44:43.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:43.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:43.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:43.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:43.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:43.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:43.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:43.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:43.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:43.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:43.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:43.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:43.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:43.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:43.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:44.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:44.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:44.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:44.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:44.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:44.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:44.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368360 2021-05-12T12:44:44.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368360 '!=' null 2021-05-12T12:44:44.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:44.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:44.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:45.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:45.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:45.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:45.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:45.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:45.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:45.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:45.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:45.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:45.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:45.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:45.367 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:45.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:45.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:45.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:45.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:45.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:45.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:45.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:45.865 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:45.865 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:46.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989369 2021-05-12T12:44:46.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989369 '!=' null 2021-05-12T12:44:46.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:46.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:46.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:46.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:46.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:46.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:46.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:46.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:46.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:46.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:46.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:46.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:46.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:46.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:46.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:47.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:47.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:47.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:47.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:47.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:47.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:48.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989369 2021-05-12T12:44:48.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989369 '!=' null 2021-05-12T12:44:48.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:48.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:48.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:48.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:48.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:48.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:48.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:48.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:48.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:48.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:48.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:48.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:48.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:48.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:48.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:49.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:49.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:49.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:49.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:49.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:49.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:49.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368361 2021-05-12T12:44:49.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368361 '!=' null 2021-05-12T12:44:49.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:49.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:49.895 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:49.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:49.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:49.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:49.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:49.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:49.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:49.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:49.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:50.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:50.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:50.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:50.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:50.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:50.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:50.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:50.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:50.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:51.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368355 2021-05-12T12:44:51.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368355 '!=' null 2021-05-12T12:44:51.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:51.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:51.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:51.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:51.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:51.268 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:51.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:51.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:51.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:51.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:51.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:51.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:51.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:51.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:51.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:52.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:52.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:52.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:52.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:52.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:52.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:52.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:52.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:52.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:52.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115670 2021-05-12T12:44:52.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115670 '!=' null 2021-05-12T12:44:52.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:52.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:52.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:52.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:52.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:52.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:52.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:52.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:52.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:52.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:52.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:53.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:53.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:53.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:53.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:53.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:53.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:53.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:53.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:53.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:53.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:53.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:53.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:53.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:54.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368314 2021-05-12T12:44:54.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368314 '!=' null 2021-05-12T12:44:54.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:54.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:54.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:54.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:54.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:54.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:54.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:54.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:54.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:54.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:54.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:54.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:54.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:54.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:54.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:54.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:54.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:54.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:54.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:54.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:54.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:54.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:54.973 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:54.974 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:55.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368314 2021-05-12T12:44:55.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368314 '!=' null 2021-05-12T12:44:55.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:55.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:55.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:55.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:55.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:55.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:55.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:55.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:55.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:55.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:55.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:55.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:55.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:55.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:55.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:56.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:56.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:56.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:56.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:56.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:56.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:56.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:56.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:56.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:56.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989303 2021-05-12T12:44:56.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989303 '!=' null 2021-05-12T12:44:56.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:56.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:56.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:56.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:56.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:56.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:56.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:56.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:56.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:56.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:56.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:57.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:57.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:57.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:57.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:57.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:57.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:57.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:57.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:57.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:57.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:57.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:57.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:57.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:58.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:44:58.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:44:58.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:58.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:58.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:58.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:58.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:58.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:58.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:58.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:58.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:58.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:58.399 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:44:58.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:44:58.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:44:58.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:44:58.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:44:59.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:44:59.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:44:59.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:44:59.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:44:59.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:44:59.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:44:59.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:44:59.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:44:59.247 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:44:59.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T12:44:59.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T12:44:59.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:44:59.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:44:59.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:44:59.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:44:59.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:44:59.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:44:59.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:44:59.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:44:59.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:44:59.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:44:59.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:00.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:00.198 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:00.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:00.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:00.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:00.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:00.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:00.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:00.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:00.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:00.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:00.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:00.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:01.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:45:01.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:45:01.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:01.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:01.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:01.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:01.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:01.276 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:01.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:01.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:01.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:01.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:01.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:01.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:01.631 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:01.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:01.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:02.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:02.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:02.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:02.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:02.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:02.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:02.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:02.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989249 2021-05-12T12:45:02.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989249 '!=' null 2021-05-12T12:45:02.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:02.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:02.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:02.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:02.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:02.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:02.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:02.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:02.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:02.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:02.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:03.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:03.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:03.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:03.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:03.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:03.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:03.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:03.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:03.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:03.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:04.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241937 2021-05-12T12:45:04.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241937 '!=' null 2021-05-12T12:45:04.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:04.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:04.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:04.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:04.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:04.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:04.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:04.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:04.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:04.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:04.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:04.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:04.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:04.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:05.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:05.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:05.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:05.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:05.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:05.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:05.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:05.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:05.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:05.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241937 2021-05-12T12:45:05.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241937 '!=' null 2021-05-12T12:45:05.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:05.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:05.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:05.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:05.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:05.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:05.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:05.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:05.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:05.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:05.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:06.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:06.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:06.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:06.094 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:06.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:06.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:06.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:06.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:06.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:06.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:06.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:06.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:06.584 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:07.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862913 2021-05-12T12:45:07.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862913 '!=' null 2021-05-12T12:45:07.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:07.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:07.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:07.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:07.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:07.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:07.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:07.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:07.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:07.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:07.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:07.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:07.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:07.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:07.517 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:08.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:08.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:08.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:08.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:08.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:08.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:08.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:08.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:08.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:08.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241945 2021-05-12T12:45:08.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241945 '!=' null 2021-05-12T12:45:08.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:08.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:08.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:08.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:08.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:08.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:08.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:08.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:08.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:08.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:08.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:09.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:09.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:09.020 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:09.020 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:09.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:09.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:09.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:09.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:09.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:09.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:09.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:09.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:09.504 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:09.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241936 2021-05-12T12:45:09.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241936 '!=' null 2021-05-12T12:45:09.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:09.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:09.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:10.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:10.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:10.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:10.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:10.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:10.092 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:10.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:10.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:10.550 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:10.551 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:10.552 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:10.552 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:11.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:11.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:11.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:11.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:11.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:11.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:11.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:11.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:11.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:11.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241936 2021-05-12T12:45:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241936 '!=' null 2021-05-12T12:45:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:11.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:11.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:11.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:11.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:11.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:11.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:11.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:11.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:11.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:11.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:11.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:11.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:11.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:12.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:12.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:12.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:12.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:12.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:12.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:12.451 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:12.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:12.452 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:12.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736562 2021-05-12T12:45:12.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736562 '!=' null 2021-05-12T12:45:12.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:12.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:12.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:13.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:13.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:13.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:13.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:13.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:13.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:13.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:13.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:13.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:13.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:13.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:13.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:13.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:13.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:13.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:13.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:13.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:13.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:13.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:13.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:13.968 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:14.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115639 2021-05-12T12:45:14.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115639 '!=' null 2021-05-12T12:45:14.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:14.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:14.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:14.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:14.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:14.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:14.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:14.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:14.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:14.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:14.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:14.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:14.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:14.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:14.899 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:15.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:15.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:15.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:15.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:15.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:15.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:15.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:15.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:15.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:15.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862931 2021-05-12T12:45:15.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862931 '!=' null 2021-05-12T12:45:15.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:15.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:15.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:15.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:15.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:15.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:15.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:15.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:16.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:16.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:16.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:16.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:16.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:16.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:16.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:16.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:16.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:16.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:16.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:16.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:16.816 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:17.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862931 2021-05-12T12:45:17.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862931 '!=' null 2021-05-12T12:45:17.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:17.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:17.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:17.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:17.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:17.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:17.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:17.411 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:17.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:17.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:17.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:17.889 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:17.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:17.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:17.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:18.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:18.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:18.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:18.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:18.463 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:18.463 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:18.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241964 2021-05-12T12:45:18.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241964 '!=' null 2021-05-12T12:45:18.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:18.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:18.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:19.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:19.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:19.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:19.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:19.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:19.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:19.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:19.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:19.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:19.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:19.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:19.416 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:19.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:19.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:19.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:19.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:19.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:19.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:19.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:19.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:19.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241955 2021-05-12T12:45:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241955 '!=' null 2021-05-12T12:45:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:20.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:20.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:20.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:20.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:20.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:20.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:20.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:20.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:20.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:20.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:20.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:20.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:20.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:20.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:21.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:21.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:21.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:21.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:21.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:21.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:21.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:21.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:21.431 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:21.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862911 2021-05-12T12:45:21.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862911 '!=' null 2021-05-12T12:45:21.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:21.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:21.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:22.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:22.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:22.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:22.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:22.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:22.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:22.010 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:22.011 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:22.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:22.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:22.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:22.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:22.856 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:22.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:22.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:22.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:22.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:22.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:22.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:22.858 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:22.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:23.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862911 2021-05-12T12:45:23.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862911 '!=' null 2021-05-12T12:45:23.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:23.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:23.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:23.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:23.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:23.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:23.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:23.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:23.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:23.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:23.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:23.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:23.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:23.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:23.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:24.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:24.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:24.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:24.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:24.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:24.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:24.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:24.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:24.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:24.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368240 2021-05-12T12:45:24.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368240 '!=' null 2021-05-12T12:45:24.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:24.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:24.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:24.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:24.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:24.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:24.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:24.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:24.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:24.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:24.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:25.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:25.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:25.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:25.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:25.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:25.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:25.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:25.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:25.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:25.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:25.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:25.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:25.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:26.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115497 2021-05-12T12:45:26.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115497 '!=' null 2021-05-12T12:45:26.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:26.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:26.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:26.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:26.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:26.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:26.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:26.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:26.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:26.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:26.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:26.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:26.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:26.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:26.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:27.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:27.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:27.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:27.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:27.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:27.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:27.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:27.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:27.162 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:27.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115497 2021-05-12T12:45:27.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115497 '!=' null 2021-05-12T12:45:27.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:27.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:27.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:27.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:27.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:27.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:27.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:27.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:27.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:27.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:27.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:28.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:28.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:28.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:28.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:28.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:28.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:28.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:28.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:28.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:28.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:28.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:28.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:28.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:29.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368232 2021-05-12T12:45:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368232 '!=' null 2021-05-12T12:45:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:29.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:29.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:29.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:29.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:29.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:29.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:29.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:29.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:29.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:29.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:29.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:29.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:29.538 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:30.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:30.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:30.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:30.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:30.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:30.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:30.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:30.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:30.021 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:30.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368219 2021-05-12T12:45:30.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368219 '!=' null 2021-05-12T12:45:30.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:30.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:30.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:30.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:30.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:30.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:30.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:30.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:30.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:30.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:30.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:30.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:30.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:30.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:30.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:31.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:31.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:31.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:31.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:31.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:31.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:31.463 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:31.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:31.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:31.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989180 2021-05-12T12:45:31.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989180 '!=' null 2021-05-12T12:45:31.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:31.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:31.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:32.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:32.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:32.076 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:32.076 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:32.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:32.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:32.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:32.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:32.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:32.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:32.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:32.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:32.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:32.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:32.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:32.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:32.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:32.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:33.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989180 2021-05-12T12:45:33.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989180 '!=' null 2021-05-12T12:45:33.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:33.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:33.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:33.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:33.506 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:33.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:33.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:33.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:33.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:33.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:33.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:33.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:33.867 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:33.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:33.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:34.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:34.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:34.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:34.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:34.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:35.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368241 2021-05-12T12:45:35.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368241 '!=' null 2021-05-12T12:45:35.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:35.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:35.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:35.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:35.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:35.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:35.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:35.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:35.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:35.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:35.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:35.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:35.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:35.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:35.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:36.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:36.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:36.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:36.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:36.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:36.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:36.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6290360 2021-05-12T12:45:36.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6290360 '!=' null 2021-05-12T12:45:36.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:36.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:36.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:36.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:36.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:36.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:36.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:36.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:36.818 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:36.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:36.819 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:37.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:37.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:37.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:37.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:37.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:37.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:37.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:37.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:37.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:37.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:37.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:37.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:37.655 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:38.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862847 2021-05-12T12:45:38.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862847 '!=' null 2021-05-12T12:45:38.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:38.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:38.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:38.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:38.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:38.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:38.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:38.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:38.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:38.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:38.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:38.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:38.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:38.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:38.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:39.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:39.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862847 2021-05-12T12:45:39.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862847 '!=' null 2021-05-12T12:45:39.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:39.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:39.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:39.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:39.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:39.700 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:39.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:39.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:39.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:39.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:39.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:40.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:40.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:40.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:40.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:40.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:40.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:40.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:40.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:40.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:40.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:40.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:40.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:40.623 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:41.088 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3931424 2021-05-12T12:45:41.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3931424 '!=' null 2021-05-12T12:45:41.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:41.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:41.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:41.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:41.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:41.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:41.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:41.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:41.191 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:41.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:41.192 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:41.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:45:41.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:41.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:41.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T12:45:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:45:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:42.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:42.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:42.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:42.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:42.038 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:42.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241927 2021-05-12T12:45:42.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241927 '!=' null 2021-05-12T12:45:42.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:42.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:42.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:42.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:42.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:42.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:42.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:42.632 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:42.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:42.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:42.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:42.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:42.977 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:42.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:42.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:43.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:43.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 0 2021-05-12T12:45:43.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:45:43.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=1 2021-05-12T12:45:43.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:43.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:43.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:43.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:43.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:43.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:43.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:43.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:43.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:43.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:43.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:43.958 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:43.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:43.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:44.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:44.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:44.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:44.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:44.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3742454 2021-05-12T12:45:44.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3742454 '!=' null 2021-05-12T12:45:44.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:44.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:44.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:45.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:45.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:45.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:45.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:45.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:45.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:45.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:45.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:45.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:45.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:45.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:45.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:45.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:45.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:45.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:45.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:46.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3145176 2021-05-12T12:45:46.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3145176 '!=' null 2021-05-12T12:45:46.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:46.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:46.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:46.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:46.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:46.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:46.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:46.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:46.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:46.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:46.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:46.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:46.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:46.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:46.832 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:47.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:47.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:47.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:47.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:47.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:47.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:47.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:47.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:47.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:47.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3145176 2021-05-12T12:45:47.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3145176 '!=' null 2021-05-12T12:45:47.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:47.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:47.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:47.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:47.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:47.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:47.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:47.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:47.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:47.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:47.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:48.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:48.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:48.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:48.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:48.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:48.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:48.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:48.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:48.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:48.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:48.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:48.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:48.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:49.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2631775 2021-05-12T12:45:49.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2631775 '!=' null 2021-05-12T12:45:49.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:49.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:49.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:49.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:49.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:49.311 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:49.312 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:49.312 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:49.312 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:49.312 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:49.313 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:49.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:49.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:49.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:49.669 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:50.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:50.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:50.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:50.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:50.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:50.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:50.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:50.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:50.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:50.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621018 2021-05-12T12:45:50.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621018 '!=' null 2021-05-12T12:45:50.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:50.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:50.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:50.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:50.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:50.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:50.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:50.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:50.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:50.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:50.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:51.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:51.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:51.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:51.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:51.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:51.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:51.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:51.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:51.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:51.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:51.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:51.559 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:51.559 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:52.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368364 2021-05-12T12:45:52.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368364 '!=' null 2021-05-12T12:45:52.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:52.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:52.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:52.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:52.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:52.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:52.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:52.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:52.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:52.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:52.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:52.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:52.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:52.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:52.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:52.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:52.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:52.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:52.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:52.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:52.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:52.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:52.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:52.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:53.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368364 2021-05-12T12:45:53.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368364 '!=' null 2021-05-12T12:45:53.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:53.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:53.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:53.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:53.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:53.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:53.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:53.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:53.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:53.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:53.548 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:53.896 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:53.897 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:53.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:53.898 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:54.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:54.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:54.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:54.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:54.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:54.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:54.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:54.373 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:54.373 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:54.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494694 2021-05-12T12:45:54.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494694 '!=' null 2021-05-12T12:45:54.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:54.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:54.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:54.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:54.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:54.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:54.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:54.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:54.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:54.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:55.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:55.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:55.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:55.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:55.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:55.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:55.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:55.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:55.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:55.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:55.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:55.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:55.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:56.290 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T12:45:56.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T12:45:56.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:56.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:56.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:56.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:56.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:56.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:56.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:56.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:56.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:56.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:56.394 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:56.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:56.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:56.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:56.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:57.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:57.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:57.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:57.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:57.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:57.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:57.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:57.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:57.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:57.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T12:45:57.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T12:45:57.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:57.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:57.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:57.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:57.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:57.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:57.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:57.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:57.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:57.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:57.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:58.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:58.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:58.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:58.165 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:45:58.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:45:58.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:45:58.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:45:58.650 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:45:58.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:45:58.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:45:58.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:45:58.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:45:58.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:45:59.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621003 2021-05-12T12:45:59.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621003 '!=' null 2021-05-12T12:45:59.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:45:59.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:45:59.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:45:59.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:45:59.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:45:59.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:45:59.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:45:59.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:45:59.243 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:45:59.244 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:45:59.244 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:45:59.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:45:59.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:45:59.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:45:59.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:00.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:00.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:00.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:00.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:00.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:00.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:00.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:00.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:00.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:00.561 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621007 2021-05-12T12:46:00.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621007 '!=' null 2021-05-12T12:46:00.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:00.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:00.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:00.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:00.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:00.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:00.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:00.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:00.666 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:00.666 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:00.667 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:01.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:01.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:01.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:01.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:01.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:01.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:01.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:01.502 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:02.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368347 2021-05-12T12:46:02.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368347 '!=' null 2021-05-12T12:46:02.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:02.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:02.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:02.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:02.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:02.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:02.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:02.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:02.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:02.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:02.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:02.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:02.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:02.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:02.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:02.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:02.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:02.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:02.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:02.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:02.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368347 2021-05-12T12:46:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368347 '!=' null 2021-05-12T12:46:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:03.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:03.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:03.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:03.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:03.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:03.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:03.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:03.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:03.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:03.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:03.938 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:03.939 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:03.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:03.940 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:04.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:04.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:04.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:04.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:04.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:04.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:04.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:04.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:04.414 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:04.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621008 2021-05-12T12:46:04.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621008 '!=' null 2021-05-12T12:46:04.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:04.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:04.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:04.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:04.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:05.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:05.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:05.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:05.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:05.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:05.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:05.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:05.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:05.363 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:05.364 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:05.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:05.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:05.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:05.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:05.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:05.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:05.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:05.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:05.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:06.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494683 2021-05-12T12:46:06.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494683 '!=' null 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:06.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:06.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:06.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:06.459 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:06.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:06.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:06.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:06.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:06.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:06.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:07.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:07.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:07.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:07.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:07.295 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:07.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:07.296 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:07.296 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:07.297 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:07.790 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494683 2021-05-12T12:46:07.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494683 '!=' null 2021-05-12T12:46:07.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:07.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:07.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:07.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:07.891 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:07.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:07.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:07.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:07.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:07.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:07.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:08.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:08.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:08.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:08.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:09.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:09.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:09.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:09.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:09.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:09.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:09.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:09.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:09.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:09.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621033 2021-05-12T12:46:09.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621033 '!=' null 2021-05-12T12:46:09.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:09.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:09.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:09.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:09.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:09.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:09.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:09.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:09.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:09.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:09.630 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:09.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:09.986 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:09.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:09.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:10.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:10.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:10.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:10.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:10.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:10.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:10.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:10.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:10.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:10.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621040 2021-05-12T12:46:10.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621040 '!=' null 2021-05-12T12:46:10.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:10.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:10.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:11.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:11.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:11.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:11.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:11.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:11.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:11.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:11.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:11.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:11.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:11.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:11.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:11.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:11.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:11.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:11.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:11.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:11.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:11.918 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:11.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:11.919 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:12.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368383 2021-05-12T12:46:12.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368383 '!=' null 2021-05-12T12:46:12.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:12.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:12.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:12.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:12.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:12.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:12.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:12.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:12.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:12.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:12.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:12.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:12.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:12.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:12.870 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:13.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:13.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:13.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:13.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:13.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:13.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:13.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:13.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:13.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:13.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368383 2021-05-12T12:46:13.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368383 '!=' null 2021-05-12T12:46:13.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:13.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:13.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:13.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:13.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:13.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:13.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:13.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:13.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:13.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:13.951 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:14.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:14.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:14.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:14.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:14.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:14.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:14.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:14.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:14.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:14.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:14.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:14.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:14.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:15.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621020 2021-05-12T12:46:15.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621020 '!=' null 2021-05-12T12:46:15.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:15.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:15.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:15.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:15.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:15.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:15.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:15.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:15.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:15.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:15.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:15.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:15.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:15.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:15.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:16.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:16.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:16.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:16.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:16.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:16.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:16.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:16.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:16.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:16.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494701 2021-05-12T12:46:16.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494701 '!=' null 2021-05-12T12:46:16.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:16.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:16.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:16.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:16.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:16.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:16.808 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:16.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:16.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:16.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:16.809 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:17.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:17.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:17.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:17.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:17.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:17.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:17.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:17.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:17.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:17.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:17.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:17.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:17.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:18.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621015 2021-05-12T12:46:18.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621015 '!=' null 2021-05-12T12:46:18.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:18.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:18.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:18.250 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:18.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:18.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:18.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:18.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:18.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:18.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:18.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:18.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:18.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:18.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:18.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:19.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:19.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:19.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:19.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:19.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:19.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:19.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:19.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:19.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:19.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621015 2021-05-12T12:46:19.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621015 '!=' null 2021-05-12T12:46:19.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:19.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:19.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:19.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:19.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:19.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:19.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:19.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:19.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:19.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:19.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:20.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:20.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:20.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:20.071 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:20.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:20.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:20.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:20.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:20.563 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:20.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:20.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:20.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:20.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:21.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620995 2021-05-12T12:46:21.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620995 '!=' null 2021-05-12T12:46:21.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:21.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:21.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:21.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:21.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:21.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:21.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:21.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:21.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:21.150 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:21.151 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:21.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:21.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:21.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:21.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:21.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:21.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:21.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:21.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:22.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368298 2021-05-12T12:46:22.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368298 '!=' null 2021-05-12T12:46:22.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:22.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:22.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:22.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:22.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:22.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:22.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:22.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:22.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:22.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:22.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:22.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:22.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:22.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:22.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:23.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:23.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:23.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:23.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:23.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:23.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:23.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368298 2021-05-12T12:46:23.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368298 '!=' null 2021-05-12T12:46:23.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:23.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:23.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:24.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:24.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:24.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:24.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:24.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:24.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:24.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:24.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:24.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:24.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:24.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:24.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:24.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:24.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:24.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:24.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:24.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:24.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:24.888 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:25.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620958 2021-05-12T12:46:25.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620958 '!=' null 2021-05-12T12:46:25.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:25.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:25.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:25.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:25.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:25.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:25.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:25.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:25.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:25.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:25.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:25.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:25.838 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:25.838 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:25.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:26.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:26.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:26.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:26.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:26.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:26.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:26.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:26.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:26.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:26.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368264 2021-05-12T12:46:26.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368264 '!=' null 2021-05-12T12:46:26.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:26.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:26.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:26.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:26.899 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:26.900 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:26.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:26.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:26.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:26.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:26.901 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:27.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:27.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:27.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:27.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:27.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:27.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:27.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:27.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:27.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:27.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:27.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:27.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:27.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:28.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494594 2021-05-12T12:46:28.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494594 '!=' null 2021-05-12T12:46:28.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:28.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:28.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:28.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:28.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:28.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:28.337 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:28.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:28.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:28.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:28.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:28.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:28.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:28.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:28.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:29.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:29.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:29.187 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:29.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:29.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:29.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:29.188 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:29.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:29.189 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:29.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494594 2021-05-12T12:46:29.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494594 '!=' null 2021-05-12T12:46:29.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:29.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:29.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:29.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:29.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:29.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:29.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:29.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:29.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:29.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:29.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:30.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:30.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:30.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:30.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:30.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:30.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:30.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:30.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:30.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:30.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:30.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:30.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:30.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:31.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494610 2021-05-12T12:46:31.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494610 '!=' null 2021-05-12T12:46:31.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:31.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:31.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:31.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:31.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:31.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:31.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:31.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:31.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:31.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:31.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:31.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:31.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:31.579 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:31.580 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:32.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:32.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:32.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:32.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:32.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:32.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:32.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:32.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:32.062 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:32.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368259 2021-05-12T12:46:32.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368259 '!=' null 2021-05-12T12:46:32.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:32.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:32.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:32.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:32.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:32.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:32.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:32.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:32.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:32.648 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:32.649 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:32.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:32.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:32.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:32.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:33.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:33.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:33.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:33.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:33.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:33.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:33.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:33.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:33.629 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:34.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620947 2021-05-12T12:46:34.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620947 '!=' null 2021-05-12T12:46:34.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:34.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:34.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:34.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:34.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:34.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:34.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:34.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:34.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:34.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:34.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:34.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:34.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:35.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:35.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:35.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:35.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:35.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:35.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:35.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:35.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:35.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:35.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620947 2021-05-12T12:46:35.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620947 '!=' null 2021-05-12T12:46:35.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:35.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:35.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:35.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:35.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:35.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:35.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:35.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:35.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:35.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:35.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:36.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:36.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:36.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:36.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:36.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:36.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:36.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:36.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:36.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:36.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:36.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:36.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:36.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:37.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T12:46:37.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T12:46:37.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:37.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:37.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:37.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:37.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:37.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:37.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:37.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:37.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:37.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:37.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:37.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:37.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:37.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:37.628 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:38.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:38.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:38.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:38.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:38.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:38.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:38.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:38.111 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:38.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:38.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620968 2021-05-12T12:46:38.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620968 '!=' null 2021-05-12T12:46:38.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:38.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:38.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:38.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:38.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:38.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:38.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:38.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:38.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:38.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:38.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:39.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:39.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:39.079 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:39.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:39.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:39.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:39.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:39.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:39.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:39.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:39.672 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:39.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:39.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:40.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620974 2021-05-12T12:46:40.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620974 '!=' null 2021-05-12T12:46:40.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:40.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:40.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:40.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:40.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:40.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:40.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:40.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:40.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:40.266 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:40.267 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:40.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:40.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:40.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:40.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:41.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:41.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:41.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:41.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:41.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:41.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:41.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:41.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:41.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:41.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620974 2021-05-12T12:46:41.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620974 '!=' null 2021-05-12T12:46:41.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:41.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:41.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:41.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:41.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:41.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:41.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:41.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:41.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:41.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:41.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:42.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:42.051 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:42.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:42.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:42.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:42.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:42.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:42.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:42.545 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:42.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:42.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:42.546 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:42.547 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:43.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494622 2021-05-12T12:46:43.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494622 '!=' null 2021-05-12T12:46:43.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:43.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:43.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:43.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:43.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:43.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:43.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:43.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:43.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:43.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:43.158 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:43.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:43.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:43.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:43.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:44.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:44.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:44.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:44.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:44.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:44.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:44.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:44.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:44.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:44.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620983 2021-05-12T12:46:44.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620983 '!=' null 2021-05-12T12:46:44.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:44.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:44.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:44.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:44.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:44.599 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:44.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:44.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:44.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:44.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:44.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:44.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:44.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:44.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:45.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:45.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:45.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:45.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:45.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:45.454 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:45.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:45.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:45.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:45.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368343 2021-05-12T12:46:45.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368343 '!=' null 2021-05-12T12:46:45.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:45.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:45.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:46.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:46.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:46.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:46.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:46.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:46.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:46.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:46.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:46.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:46.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:46.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:46.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:46.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:46.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:46.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:46.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:46.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:46.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:46.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:46.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:46.918 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:47.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368343 2021-05-12T12:46:47.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368343 '!=' null 2021-05-12T12:46:47.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:47.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:47.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:47.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:47.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:47.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:47.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:47.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:47.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:47.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:47.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:47.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:47.859 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:47.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:47.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:48.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:48.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:48.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:48.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:48.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:48.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:48.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:48.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:48.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:48.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621007 2021-05-12T12:46:48.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621007 '!=' null 2021-05-12T12:46:48.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:48.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:48.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:48.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:48.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:48.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:48.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:48.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:48.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:48.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:48.929 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:49.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:49.293 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:49.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:49.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:49.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:49.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:49.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:49.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:49.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:49.778 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:49.778 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:50.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620999 2021-05-12T12:46:50.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620999 '!=' null 2021-05-12T12:46:50.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:50.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:50.270 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:50.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:50.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:50.371 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:50.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:50.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:50.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:50.372 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:50.373 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:50.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:50.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:50.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:50.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:51.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:51.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:51.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:51.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:51.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:51.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:51.219 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:51.220 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:51.220 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:51.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620999 2021-05-12T12:46:51.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620999 '!=' null 2021-05-12T12:46:51.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:51.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:51.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:51.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:51.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:51.827 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:51.827 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:51.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:51.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:51.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:51.828 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:52.186 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:52.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:52.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:52.187 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:52.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:52.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:52.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:52.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:52.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:52.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:52.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:52.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:52.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:53.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368320 2021-05-12T12:46:53.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368320 '!=' null 2021-05-12T12:46:53.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:53.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:53.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:53.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:53.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:53.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:53.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:53.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:53.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:53.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:53.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:53.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:53.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:53.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:54.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:54.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:54.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:54.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:54.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:54.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:54.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:54.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:54.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494643 2021-05-12T12:46:54.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494643 '!=' null 2021-05-12T12:46:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:54.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:54.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:54.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:54.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:54.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:54.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:54.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:54.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:54.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:55.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:55.035 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:55.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:55.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:55.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:55.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:55.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:55.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:55.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:55.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:55.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:55.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:55.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:56.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368319 2021-05-12T12:46:56.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368319 '!=' null 2021-05-12T12:46:56.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:56.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:56.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:56.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:56.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:56.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:56.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:56.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:56.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:56.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:56.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:56.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:56.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:56.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:56.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:56.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:56.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:56.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:56.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:56.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:56.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:56.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:56.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:56.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:57.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368319 2021-05-12T12:46:57.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368319 '!=' null 2021-05-12T12:46:57.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:57.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:57.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:57.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:57.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:57.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:57.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:57.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:57.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:57.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:57.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:57.894 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:57.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:57.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:57.895 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:58.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:58.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:58.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:58.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:58.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:58.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:58.458 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:58.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:58.458 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:46:58.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620974 2021-05-12T12:46:58.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620974 '!=' null 2021-05-12T12:46:58.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:46:58.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:46:58.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:46:59.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:46:59.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:46:59.041 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:46:59.041 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:46:59.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:46:59.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:46:59.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:46:59.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:46:59.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:46:59.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:46:59.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:46:59.403 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:46:59.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:46:59.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:46:59.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:46:59.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:46:59.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:46:59.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:46:59.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:46:59.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:46:59.887 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:00.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620961 2021-05-12T12:47:00.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620961 '!=' null 2021-05-12T12:47:00.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:00.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:00.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:00.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:00.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:00.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:00.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:00.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:00.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:00.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:00.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:00.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:00.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:00.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:00.834 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:01.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:01.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:01.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:01.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:01.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:01.315 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:01.316 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:01.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:01.317 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:01.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620961 2021-05-12T12:47:01.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620961 '!=' null 2021-05-12T12:47:01.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:01.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:01.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:01.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:01.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:01.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:01.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:01.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:01.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:01.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:01.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:02.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:02.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:02.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:02.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:02.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:02.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:02.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:02.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:02.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:02.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:02.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:02.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:02.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:03.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368266 2021-05-12T12:47:03.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368266 '!=' null 2021-05-12T12:47:03.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:03.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:03.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:03.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:03.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:03.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:03.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:03.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:03.374 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:03.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:03.375 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:03.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:03.726 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:03.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:03.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:04.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:04.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:04.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:04.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:04.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:04.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:04.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:04.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:04.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:04.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620967 2021-05-12T12:47:04.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620967 '!=' null 2021-05-12T12:47:04.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:04.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:04.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:04.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:04.818 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:04.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:04.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:04.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:04.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:04.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:04.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:05.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:05.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:05.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:05.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:05.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:05.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:05.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:05.654 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:05.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:05.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:05.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:05.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:05.656 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:06.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494637 2021-05-12T12:47:06.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494637 '!=' null 2021-05-12T12:47:06.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:06.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:06.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:06.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:06.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:06.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:06.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:06.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:06.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:06.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:06.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:06.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:06.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:06.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:06.591 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:07.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:07.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:07.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:07.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:07.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:07.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:07.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:07.138 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:07.139 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:07.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494637 2021-05-12T12:47:07.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494637 '!=' null 2021-05-12T12:47:07.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:07.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:07.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:07.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:07.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:07.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:07.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:07.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:07.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:07.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:07.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:08.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:08.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:08.140 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:08.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:08.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:08.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:08.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:08.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:08.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:08.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:08.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:08.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:08.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:09.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620966 2021-05-12T12:47:09.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620966 '!=' null 2021-05-12T12:47:09.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:09.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:09.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:09.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:09.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:09.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:09.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:09.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:09.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:09.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:09.227 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:09.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:09.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:09.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:09.581 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:10.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:10.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:10.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:10.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:10.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:10.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:10.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:10.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:10.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:10.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620978 2021-05-12T12:47:10.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620978 '!=' null 2021-05-12T12:47:10.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:10.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:10.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:10.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:10.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:10.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:10.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:10.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:10.633 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:10.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:10.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:10.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:10.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:10.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:10.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:11.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:11.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:11.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:11.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:11.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:11.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:11.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:11.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:11.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:47:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:47:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:11.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:11.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:12.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:12.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:12.068 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:12.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:12.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:12.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:12.450 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:12.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:12.451 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:12.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:12.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:12.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:12.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:12.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:13.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T12:47:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T12:47:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:13.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:13.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:13.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:13.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:13.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:13.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:13.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:13.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:13.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:13.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:13.855 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:13.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:13.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:14.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:14.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:14.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:14.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:14.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:14.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:14.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:14.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:14.332 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:14.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620992 2021-05-12T12:47:14.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620992 '!=' null 2021-05-12T12:47:14.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:14.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:14.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:14.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:14.922 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:14.923 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:14.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:14.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:14.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:14.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:14.924 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:15.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:15.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:15.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:15.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:15.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:15.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:15.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:15.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:15.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:15.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:15.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:16.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368329 2021-05-12T12:47:16.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368329 '!=' null 2021-05-12T12:47:16.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:16.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:16.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:16.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:16.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:16.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:16.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:16.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:16.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:16.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:16.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:16.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:16.724 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:16.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:16.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:17.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:17.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:17.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:17.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:17.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:17.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:17.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:17.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:17.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:17.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368329 2021-05-12T12:47:17.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368329 '!=' null 2021-05-12T12:47:17.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:17.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:17.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:17.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:17.783 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:17.783 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:17.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:17.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:17.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:17.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:18.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:18.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:18.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:18.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:18.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:18.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:18.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:18.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:18.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:18.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:18.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:18.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:18.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:19.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494663 2021-05-12T12:47:19.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494663 '!=' null 2021-05-12T12:47:19.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:19.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:19.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:19.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:19.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:19.203 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:19.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:19.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:19.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:19.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:19.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:19.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:19.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:19.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:19.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:20.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:20.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:20.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:20.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:20.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:20.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:20.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:20.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:20.048 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:20.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494656 2021-05-12T12:47:20.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494656 '!=' null 2021-05-12T12:47:20.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:20.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:20.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:20.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:20.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:20.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:20.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:20.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:20.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:20.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:20.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:21.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:21.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:21.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:21.042 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:21.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:21.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:21.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:21.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:21.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:21.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:21.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:21.520 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:21.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:21.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368321 2021-05-12T12:47:21.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368321 '!=' null 2021-05-12T12:47:21.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:21.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:21.996 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:22.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:22.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:22.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:22.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:22.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:22.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:22.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:22.098 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:22.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:22.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:22.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:22.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:22.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:22.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:22.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:22.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:22.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:22.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:22.925 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:22.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:22.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:23.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368321 2021-05-12T12:47:23.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368321 '!=' null 2021-05-12T12:47:23.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:23.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:23.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:23.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:23.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:23.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:23.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:23.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:23.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:23.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:23.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:23.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:23.862 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:23.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:23.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:24.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:24.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:24.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:24.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:24.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:24.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:24.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:24.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:24.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:25.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620984 2021-05-12T12:47:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620984 '!=' null 2021-05-12T12:47:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:25.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:25.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:25.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:25.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:25.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:25.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:25.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:25.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:25.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:25.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:25.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:25.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:25.654 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:26.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:26.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:26.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:26.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:26.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:26.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:26.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:26.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:26.143 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:26.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368330 2021-05-12T12:47:26.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368330 '!=' null 2021-05-12T12:47:26.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:26.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:26.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:26.731 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:26.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:26.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:26.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:26.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:26.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:26.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:26.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:27.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:27.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:27.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:27.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:27.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:27.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:27.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:27.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:27.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:27.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:27.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:27.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:27.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:28.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620984 2021-05-12T12:47:28.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620984 '!=' null 2021-05-12T12:47:28.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:28.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:28.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:28.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:28.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:28.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:28.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:28.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:28.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:28.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:28.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:28.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:28.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:28.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:28.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:29.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:29.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:29.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:29.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:29.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:29.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:29.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:29.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:29.019 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:29.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620984 2021-05-12T12:47:29.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620984 '!=' null 2021-05-12T12:47:29.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:29.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:29.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:29.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:29.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:29.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:29.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:29.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:29.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:29.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:29.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:30.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:30.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:30.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:30.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:30.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:30.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:30.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:30.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:30.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:30.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:30.493 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:30.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:30.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:30.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620982 2021-05-12T12:47:30.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620982 '!=' null 2021-05-12T12:47:30.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:30.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:30.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:31.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:31.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:31.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:31.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:31.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:31.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:31.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:31.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:31.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:31.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:31.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:31.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:31.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:31.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:31.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:31.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:31.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:31.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:31.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:31.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:31.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:32.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494646 2021-05-12T12:47:32.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494646 '!=' null 2021-05-12T12:47:32.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:32.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:32.422 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:32.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:32.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:32.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:32.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:32.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:32.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:32.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:32.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:32.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:32.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:32.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:32.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:33.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:33.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:33.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:33.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:33.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:33.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:33.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:33.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:33.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:33.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494646 2021-05-12T12:47:33.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494646 '!=' null 2021-05-12T12:47:33.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:33.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:33.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:33.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:33.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:33.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:33.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:33.949 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:33.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:33.950 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:34.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:34.306 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:34.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:34.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:34.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:34.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:34.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:34.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:34.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:34.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:35.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620986 2021-05-12T12:47:35.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620986 '!=' null 2021-05-12T12:47:35.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:35.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:35.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:35.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:35.368 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:35.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:35.369 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:35.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:35.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:35.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:35.370 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:35.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:35.724 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:35.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:35.725 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:36.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:36.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:36.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:36.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:36.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:36.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:36.218 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:36.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:36.219 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:36.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368340 2021-05-12T12:47:36.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368340 '!=' null 2021-05-12T12:47:36.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:36.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:36.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:36.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:36.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:36.804 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:36.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:36.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:36.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:36.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:36.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:37.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:37.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:37.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:37.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:37.628 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:37.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:37.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:37.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:37.629 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:37.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:37.630 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:37.631 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:37.631 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:38.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621008 2021-05-12T12:47:38.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621008 '!=' null 2021-05-12T12:47:38.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:38.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:38.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:38.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:38.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:38.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:38.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:38.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:38.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:38.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:38.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:38.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:38.719 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:38.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:38.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:39.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:39.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:39.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:39.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:39.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:39.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621008 2021-05-12T12:47:39.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621008 '!=' null 2021-05-12T12:47:39.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:39.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:39.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:39.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:39.789 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:39.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:39.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:39.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:39.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:39.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:39.791 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:40.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:40.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:40.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:40.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:40.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:40.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:40.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:40.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:40.635 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:40.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:40.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:40.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:40.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:41.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2621006 2021-05-12T12:47:41.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2621006 '!=' null 2021-05-12T12:47:41.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:41.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:41.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:41.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:41.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:41.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:41.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:41.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:41.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:41.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:41.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:41.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:41.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:41.561 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:41.562 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:42.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:42.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:42.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:42.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:42.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:42.037 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:42.037 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:42.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368327 2021-05-12T12:47:42.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368327 '!=' null 2021-05-12T12:47:42.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:42.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:42.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:42.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:42.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:42.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:42.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:42.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:42.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:42.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:42.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:42.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:42.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:42.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:42.988 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:43.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:43.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:43.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:43.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:43.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:43.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:43.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:43.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:43.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:43.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494673 2021-05-12T12:47:43.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494673 '!=' null 2021-05-12T12:47:43.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:43.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:43.958 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:44.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:44.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:44.059 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:44.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:44.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:44.060 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:44.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:44.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:44.416 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:44.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:44.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:44.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:44.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:44.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:44.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:44.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:44.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:44.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:44.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:44.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:44.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:45.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494673 2021-05-12T12:47:45.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494673 '!=' null 2021-05-12T12:47:45.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:45.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:45.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:45.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:45.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:45.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:45.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:45.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:45.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:45.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:45.506 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:45.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:45.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:45.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:45.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:46.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:46.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:46.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:46.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:46.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:46.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:46.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:46.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:46.368 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:46.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368351 2021-05-12T12:47:46.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368351 '!=' null 2021-05-12T12:47:46.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:46.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:46.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:46.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:46.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:46.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:46.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:46.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:46.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:46.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:46.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:47.297 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:47.298 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:47.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:47.299 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:47.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:47.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:47.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:47.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:47.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:47.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:47.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:47.811 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:47.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:48.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620995 2021-05-12T12:47:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620995 '!=' null 2021-05-12T12:47:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:48.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:48.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:48.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:48.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:48.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:48.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:48.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:48.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:48.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:48.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:48.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:48.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:48.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:49.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:49.272 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:49.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:49.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:49.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:49.273 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:49.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:49.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:49.274 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:49.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620995 2021-05-12T12:47:49.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620995 '!=' null 2021-05-12T12:47:49.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:49.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:49.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:49.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:49.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:49.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:49.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:49.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:49.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:49.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:49.854 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:50.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:50.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:50.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:50.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:50.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:50.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:50.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:50.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:50.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:50.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:50.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:50.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:50.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:51.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620998 2021-05-12T12:47:51.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620998 '!=' null 2021-05-12T12:47:51.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:51.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:51.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:51.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:51.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:51.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:51.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:51.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:51.285 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:51.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:51.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:51.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:51.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:52.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:52.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:52.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:52.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:52.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:52.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:52.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:52.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:52.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:52.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368307 2021-05-12T12:47:52.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368307 '!=' null 2021-05-12T12:47:52.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:52.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:52.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:52.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:52.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:52.704 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:52.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:52.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:52.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:52.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:52.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:53.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:53.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:53.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:53.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:53.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:53.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:53.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:53.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:53.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:53.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:53.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:53.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:53.531 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:54.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620983 2021-05-12T12:47:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620983 '!=' null 2021-05-12T12:47:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:54.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:54.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:54.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:54.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:54.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:54.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:54.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:54.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:54.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:54.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:54.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:55.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:55.010 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:55.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:55.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:55.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:55.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:55.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:55.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:55.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:55.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620983 2021-05-12T12:47:55.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620983 '!=' null 2021-05-12T12:47:55.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:55.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:55.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:55.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:55.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:55.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:55.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:55.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:55.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:55.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:55.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:56.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:56.026 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:56.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:56.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:56.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:56.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:56.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:56.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:56.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:56.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:56.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:56.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:56.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:56.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494641 2021-05-12T12:47:56.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494641 '!=' null 2021-05-12T12:47:56.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:56.994 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:56.995 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:57.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:57.094 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:57.095 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:57.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:57.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:57.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:57.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:57.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:57.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:57.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:57.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:57.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:57.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:57.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:57.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:57.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:57.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:58.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620951 2021-05-12T12:47:58.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620951 '!=' null 2021-05-12T12:47:58.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:58.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:58.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:58.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:58.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:58.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:58.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:58.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:58.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:58.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:58.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:47:58.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:47:58.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:47:58.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:47:58.875 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:47:59.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:47:59.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:47:59.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:47:59.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:47:59.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:47:59.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:47:59.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:47:59.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:47:59.362 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:47:59.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620951 2021-05-12T12:47:59.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620951 '!=' null 2021-05-12T12:47:59.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:47:59.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:47:59.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:47:59.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:47:59.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:47:59.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:47:59.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:47:59.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:47:59.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:47:59.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:47:59.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:00.299 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:00.300 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:00.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:00.301 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:00.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:00.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:00.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:00.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:00.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:00.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:00.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:00.783 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:00.783 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:01.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620955 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620955 '!=' null 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:01.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:01.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:01.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:01.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:01.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:01.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:01.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:01.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:01.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:01.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:01.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:01.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:01.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:01.733 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:02.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:02.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:02.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:02.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:02.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:02.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:02.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:02.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:02.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368267 2021-05-12T12:48:02.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368267 '!=' null 2021-05-12T12:48:02.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:02.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:02.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:02.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:02.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:02.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:02.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:02.787 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:02.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:02.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:02.788 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:03.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:03.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:03.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:03.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:03.631 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:03.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:03.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:03.632 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:03.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:03.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:03.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:03.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:03.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:04.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620965 2021-05-12T12:48:04.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620965 '!=' null 2021-05-12T12:48:04.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:04.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:04.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:04.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:04.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:04.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:04.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:04.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:04.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:04.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:04.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:04.574 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:04.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:04.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:04.575 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:05.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:05.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:05.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:05.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:05.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:05.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:05.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:05.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:05.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:05.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620965 2021-05-12T12:48:05.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620965 '!=' null 2021-05-12T12:48:05.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:05.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:05.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:05.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:05.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:05.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:05.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:05.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:05.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:05.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:05.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:05.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:05.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:05.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:06.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:06.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:06.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:06.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:06.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:06.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:06.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:06.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:06.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368264 2021-05-12T12:48:06.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368264 '!=' null 2021-05-12T12:48:06.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:06.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:06.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:07.063 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:07.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:07.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:07.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:07.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:07.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:07.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:07.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:07.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:07.436 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:07.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:07.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:07.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:07.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:07.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:07.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:07.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:07.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:07.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:07.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:07.928 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:08.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494596 2021-05-12T12:48:08.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494596 '!=' null 2021-05-12T12:48:08.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:08.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:08.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:08.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:08.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:08.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:08.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:08.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:08.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:08.528 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:08.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:08.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:08.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:08.889 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:08.890 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:09.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:09.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:09.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:09.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:09.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:09.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:09.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:09.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:09.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:09.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494602 2021-05-12T12:48:09.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494602 '!=' null 2021-05-12T12:48:09.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:09.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:09.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:09.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:09.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:09.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:09.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:09.971 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:09.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:09.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:09.972 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:10.330 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:10.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:10.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:10.331 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:10.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:10.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:10.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:10.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:10.810 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:11.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494602 2021-05-12T12:48:11.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494602 '!=' null 2021-05-12T12:48:11.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:11.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:11.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:11.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:11.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:11.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:11.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:11.386 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:11.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:11.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:11.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:11.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:11.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:11.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:11.742 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:12.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:12.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:12.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:12.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:12.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:12.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:12.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:12.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:12.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:12.718 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:48:12.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:48:12.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:12.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:12.719 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:12.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:12.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:12.820 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:12.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:12.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:12.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:12.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:12.822 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:13.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:13.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:13.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:13.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:13.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:13.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:13.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:13.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:13.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:13.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:13.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:13.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:13.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:14.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620948 2021-05-12T12:48:14.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620948 '!=' null 2021-05-12T12:48:14.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:14.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:14.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:14.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:14.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:14.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:14.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:14.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:14.235 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:14.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:14.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:14.591 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:14.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:14.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:14.592 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:15.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:15.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:15.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:15.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:15.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:15.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:15.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:15.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:15.072 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:15.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620948 2021-05-12T12:48:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620948 '!=' null 2021-05-12T12:48:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:15.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:15.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:15.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:15.657 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:15.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:15.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:15.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:15.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:16.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:16.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:16.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:16.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:16.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:16.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:16.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:16.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:16.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:16.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:16.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:16.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:16.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:17.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368265 2021-05-12T12:48:17.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368265 '!=' null 2021-05-12T12:48:17.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:17.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:17.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:17.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:17.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:17.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:17.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:17.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:17.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:17.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:17.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:17.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:17.480 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:17.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:17.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:18.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:18.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:18.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:18.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:18.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:18.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:18.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:18.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:18.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:18.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620956 2021-05-12T12:48:18.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620956 '!=' null 2021-05-12T12:48:18.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:18.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:18.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:18.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:18.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:18.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:18.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:18.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:18.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:18.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:18.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:18.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:18.960 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:18.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:18.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:19.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:19.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:19.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:19.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:19.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:19.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:19.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:19.448 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:19.449 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:19.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620973 2021-05-12T12:48:20.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620973 '!=' null 2021-05-12T12:48:20.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:20.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:20.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:20.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:20.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:20.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:20.080 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:20.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:20.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:20.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:20.081 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:20.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:20.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:20.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:20.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:20.914 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:20.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:20.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:20.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:20.916 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:20.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:20.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:21.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620973 2021-05-12T12:48:21.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620973 '!=' null 2021-05-12T12:48:21.413 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:21.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:21.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:21.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:21.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:21.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:21.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:21.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:21.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:21.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:21.516 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:21.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:21.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:21.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:21.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:22.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:22.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:22.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:22.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:22.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:22.432 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:23.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494656 2021-05-12T12:48:23.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494656 '!=' null 2021-05-12T12:48:23.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:23.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:23.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:23.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:23.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:23.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:23.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:23.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:23.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:23.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:23.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:23.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:23.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:23.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:23.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:24.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:24.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:24.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:24.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:24.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:24.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:24.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:24.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:24.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:24.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620961 2021-05-12T12:48:24.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620961 '!=' null 2021-05-12T12:48:24.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:24.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:24.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:24.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:24.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:24.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:24.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:24.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:24.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:24.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:24.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:25.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:25.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:25.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:25.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:25.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:25.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:25.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:25.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:25.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:25.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:25.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:25.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:25.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:26.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:48:26.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:48:26.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:26.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:26.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:26.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:26.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:26.129 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:26.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:26.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:26.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:26.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:26.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:26.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:26.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:26.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:26.487 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:26.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:26.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:26.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:26.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:26.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:26.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:26.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:26.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:26.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:27.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368258 2021-05-12T12:48:27.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368258 '!=' null 2021-05-12T12:48:27.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:27.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:27.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:27.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:27.541 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:27.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:27.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:27.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:27.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:27.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:27.543 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:27.934 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:27.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:27.935 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:27.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:28.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:28.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:28.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:28.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:28.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:28.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:28.419 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:28.420 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:28.420 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:28.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620964 2021-05-12T12:48:28.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620964 '!=' null 2021-05-12T12:48:28.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:28.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:28.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:29.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:29.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:29.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:29.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:29.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:29.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:29.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:29.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:29.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:29.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:29.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:29.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:29.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:29.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:29.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:29.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:29.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:29.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:29.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:29.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:29.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:30.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620968 2021-05-12T12:48:30.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620968 '!=' null 2021-05-12T12:48:30.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:30.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:30.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:30.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:30.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:30.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:30.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:30.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:30.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:30.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:30.513 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:30.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:30.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:30.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:30.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:31.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:31.402 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:31.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:31.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:31.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:31.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:31.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:31.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:31.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:31.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620968 2021-05-12T12:48:31.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620968 '!=' null 2021-05-12T12:48:31.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:31.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:31.888 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:31.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:31.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:31.989 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:31.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:31.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:31.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:31.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:31.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:32.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:32.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:32.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:32.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:32.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:32.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:32.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:32.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:32.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368279 2021-05-12T12:48:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368279 '!=' null 2021-05-12T12:48:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:33.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:33.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:33.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:33.429 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:33.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:33.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:33.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:33.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:33.781 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:33.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:33.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:33.782 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:34.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:34.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:34.261 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:34.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494628 2021-05-12T12:48:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494628 '!=' null 2021-05-12T12:48:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:34.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:34.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:34.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:35.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:35.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:35.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:35.210 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:35.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:35.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:35.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:35.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:35.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:35.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:35.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:35.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:35.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:36.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:48:36.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:48:36.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:36.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:36.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:36.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:36.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:36.315 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:36.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:36.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:36.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:36.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:36.316 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:36.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:36.676 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:36.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:36.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:37.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:37.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:37.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:37.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:37.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:37.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:37.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:37.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:37.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:37.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T12:48:37.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T12:48:37.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:37.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:37.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:37.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:37.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:37.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:37.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:37.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:37.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:37.750 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:37.750 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:38.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:38.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:38.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:38.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:38.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:38.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:38.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:38.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:38.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:38.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:39.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620974 2021-05-12T12:48:39.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620974 '!=' null 2021-05-12T12:48:39.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:39.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:39.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:39.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:39.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:39.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:39.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:39.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:39.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:39.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:39.226 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:39.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:39.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:39.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:39.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:40.231 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:40.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:40.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:40.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:40.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:40.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:40.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:40.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:40.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:40.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620957 2021-05-12T12:48:40.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620957 '!=' null 2021-05-12T12:48:40.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:40.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:40.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:40.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:40.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:40.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:40.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:40.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:40.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:40.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:40.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:41.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:41.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:41.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:41.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:41.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:41.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:41.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:41.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:41.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:41.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:41.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:41.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:41.653 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:42.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:48:42.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:48:42.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:42.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:42.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:42.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:42.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:42.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:42.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:42.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:42.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:42.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:42.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:42.592 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:42.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:42.594 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:42.594 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:43.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:43.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:43.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:43.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:43.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:43.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:43.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:43.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:43.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:43.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368289 2021-05-12T12:48:43.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368289 '!=' null 2021-05-12T12:48:43.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:43.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:43.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:43.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:43.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:43.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:43.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:43.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:43.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:43.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:43.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:44.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:44.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:44.016 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:44.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:44.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:44.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:44.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:44.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:44.602 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620943 2021-05-12T12:48:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620943 '!=' null 2021-05-12T12:48:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:45.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:45.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:45.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:45.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:45.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:45.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:45.236 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:45.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:45.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:45.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:45.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:45.593 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:45.594 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:45.594 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:46.080 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:46.081 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:46.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:46.082 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:46.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:46.083 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:46.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494599 2021-05-12T12:48:46.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494599 '!=' null 2021-05-12T12:48:46.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:46.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:46.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:46.671 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:46.672 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:46.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:46.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:46.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:46.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:46.673 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:47.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:47.026 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:47.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:47.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:47.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:47.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:47.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:47.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:47.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:47.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:47.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:47.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:47.505 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:47.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620957 2021-05-12T12:48:47.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620957 '!=' null 2021-05-12T12:48:47.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:47.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:47.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:48.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:48.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:48.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:48.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:48.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:48.087 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:48.088 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:48.088 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:48.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:48.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:48.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:48.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:48.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:48.923 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:48.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:48.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:48.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:48.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:48.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:48.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:48.925 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:49.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620957 2021-05-12T12:48:49.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620957 '!=' null 2021-05-12T12:48:49.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:49.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:49.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:49.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:49.492 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:49.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:49.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:49.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:49.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:49.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:49.495 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:49.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:49.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:49.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:49.874 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:50.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:50.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:50.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:50.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:50.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:50.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620960 2021-05-12T12:48:50.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620960 '!=' null 2021-05-12T12:48:50.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:50.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:50.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:50.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:50.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:50.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:50.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:50.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:50.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:50.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:50.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:51.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:51.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:51.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:51.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:51.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:51.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:51.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:51.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:51.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:51.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:51.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:51.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:51.793 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:52.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241972 2021-05-12T12:48:52.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241972 '!=' null 2021-05-12T12:48:52.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:52.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:52.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:52.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:52.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:52.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:52.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:52.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:52.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:52.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:52.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:52.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:52.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:52.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:52.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:53.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:53.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:53.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:53.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:53.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:53.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:53.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:53.218 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:53.218 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:53.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241972 2021-05-12T12:48:53.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241972 '!=' null 2021-05-12T12:48:53.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:53.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:53.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:53.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:53.788 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:53.789 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:53.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:53.790 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:54.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:54.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:54.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:54.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:54.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:54.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:54.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:54.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:54.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:55.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494617 2021-05-12T12:48:55.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494617 '!=' null 2021-05-12T12:48:55.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:55.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:55.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:55.352 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:55.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:55.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:55.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:55.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:55.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:55.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:55.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:55.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:55.707 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:55.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:55.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:56.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:56.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:56.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:56.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:56.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:56.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:56.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:56.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:56.202 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:56.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368324 2021-05-12T12:48:56.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368324 '!=' null 2021-05-12T12:48:56.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:56.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:56.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:56.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:56.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:56.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:56.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:56.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:56.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:56.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:56.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:57.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:57.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:57.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:57.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:57.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:57.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:57.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:57.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:57.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:57.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:57.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:57.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:57.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:58.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494624 2021-05-12T12:48:58.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494624 '!=' null 2021-05-12T12:48:58.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:58.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:58.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:58.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:58.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:58.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:58.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:58.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:58.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:58.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:58.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:48:58.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:48:58.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:48:58.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:48:58.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:48:59.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:48:59.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:48:59.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:48:59.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:48:59.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:48:59.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:48:59.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:48:59.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:48:59.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:48:59.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494624 2021-05-12T12:48:59.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494624 '!=' null 2021-05-12T12:48:59.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:48:59.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:48:59.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:48:59.725 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:48:59.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:48:59.727 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:48:59.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:48:59.728 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:48:59.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:48:59.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:48:59.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:49:00.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:49:00.085 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:49:00.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:49:00.086 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:49:00.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:49:00.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:49:00.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:49:00.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:49:00.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:49:00.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:49:00.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:49:00.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:49:00.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:49:01.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3494616 2021-05-12T12:49:01.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3494616 '!=' null 2021-05-12T12:49:01.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:49:01.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:49:01.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:49:01.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:49:01.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:49:01.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:49:01.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:49:01.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:49:01.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:49:01.315 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:49:01.315 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:49:01.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:49:01.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:49:01.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:49:01.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:49:02.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:49:02.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:49:02.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:49:02.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:49:02.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:49:02.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:49:02.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:49:02.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:49:02.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:49:02.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3931438 2021-05-12T12:49:02.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3931438 '!=' null 2021-05-12T12:49:02.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:49:02.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:49:02.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:49:02.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:49:02.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:49:02.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:49:02.705 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:49:02.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:49:02.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:49:02.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:49:02.706 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:49:03.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:49:03.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:49:03.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:49:03.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:49:03.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:49:03.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:49:03.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:49:03.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:49:03.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:49:03.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:49:03.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:49:03.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:49:03.535 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:49:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620951 2021-05-12T12:49:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620951 '!=' null 2021-05-12T12:49:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:49:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:49:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:49:04.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:49:04.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:49:04.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:49:04.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:49:04.106 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:49:04.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:49:04.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:49:04.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:49:04.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=1 2021-05-12T12:49:04.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:49:04.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:49:04.475 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:49:04.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 = 2 2021-05-12T12:49:04.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 1 '!=' 1 2021-05-12T12:49:04.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:49:04.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:49:04.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:49:04.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:49:04.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:49:04.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:49:04.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:49:05.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620951 2021-05-12T12:49:05.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620951 '!=' null 2021-05-12T12:49:05.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:49:05.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T12:49:05.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:49:05.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:49:05.521 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:49:05.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:49:05.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:49:05.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:49:05.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:49:05.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:49:05.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:49:05.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T12:49:05.871 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:49:05.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:49:05.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:49:06.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T12:49:06.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:49:06.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:49:06.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:522: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2021-05-12T12:49:06.339 INFO:tasks.workunit.client.0.smithi090.stdout:TEST PASSED 2021-05-12T12:49:06.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:526: TEST_recovery_pool_priority: echo TEST PASSED 2021-05-12T12:49:06.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: delete_pool test1 2021-05-12T12:49:06.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-05-12T12:49:06.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-05-12T12:49:06.818 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test1' does not exist 2021-05-12T12:49:06.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:530: TEST_recovery_pool_priority: delete_pool test2 2021-05-12T12:49:06.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-05-12T12:49:06.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-05-12T12:49:07.557 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test2' does not exist 2021-05-12T12:49:07.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:531: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2021-05-12T12:49:07.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:49:07.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:49:07.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:49:07.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:49:07.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:49:07.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:49:07.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: return 0 2021-05-12T12:49:07.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-05-12T12:49:07.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:49:07.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:49:07.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:49:07.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:49:07.878 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:49:07.878 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:49:07.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:49:07.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:49:07.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:49:07.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:49:07.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:49:07.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:49:07.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:49:07.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:49:07.883 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:49:07.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:49:07.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:49:07.884 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:49:07.885 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:49:07.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:49:07.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:49:07.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:49:08.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:49:08.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:08.686 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:08.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.185878 2021-05-12T12:49:08.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:49:08.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:49:08.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-05-12T12:49:08.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-05-12T12:49:08.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-05-12T12:49:08.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-05-12T12:49:08.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T12:49:08.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T12:49:08.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T12:49:08.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T12:49:08.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T12:49:08.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T12:49:08.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T12:49:08.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T12:49:08.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T12:49:08.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T12:49:08.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T12:49:08.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T12:49:08.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T12:49:08.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T12:49:08.694 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T12:49:08.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:49:08.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T12:49:08.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T12:49:08.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T12:49:08.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T12:49:08.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T12:49:08.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T12:49:08.697 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T12:49:08.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:08.698 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:08.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.185878 2021-05-12T12:49:08.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T12:49:08.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T12:49:08.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-05-12T12:49:08.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T12:49:08.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:08.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:08.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.185878 2021-05-12T12:49:08.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T12:49:08.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T12:49:08.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T12:49:08.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-05-12T12:49:08.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_priority td/osd-recovery-prio 2021-05-12T12:49:08.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:44: TEST_recovery_priority: local dir=td/osd-recovery-prio 2021-05-12T12:49:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:45: TEST_recovery_priority: local pools=10 2021-05-12T12:49:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:46: TEST_recovery_priority: local OSDS=5 2021-05-12T12:49:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local max_tries=10 2021-05-12T12:49:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2021-05-12T12:49:08.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-05-12T12:49:08.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T12:49:08.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T12:49:08.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T12:49:08.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-05-12T12:49:08.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-05-12T12:49:09.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T12:49:09.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:09.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:09.075 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:09.075 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:09.075 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:09.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:09.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T12:49:09.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T12:49:09.344 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T12:49:09.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:49:09.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:49:09.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T12:49:09.345 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T12:49:09.346 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:49:09.346 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:49:09.346 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:49:09.346 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:49:09.346 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:09.347 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:09.347 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-mon.a.asok 2021-05-12T12:49:09.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:49:09.347 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185878/ceph-mon.a.asok config get fsid 2021-05-12T12:49:09.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T12:49:09.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T12:49:09.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T12:49:09.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T12:49:09.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T12:49:09.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T12:49:09.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T12:49:09.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T12:49:09.441 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:49:09.441 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:09.441 INFO:tasks.workunit.client.0.smithi090.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:09.441 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-mon.a.asok 2021-05-12T12:49:09.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T12:49:09.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185878/ceph-mon.a.asok config get mon_host 2021-05-12T12:49:09.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2021-05-12T12:49:09.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-05-12T12:49:09.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T12:49:09.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T12:49:09.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T12:49:09.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-05-12T12:49:09.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T12:49:10.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T12:49:10.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:10.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:10.405 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:10.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:10.406 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:10.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:10.406 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:49:10.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T12:49:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:51: TEST_recovery_priority: export CEPH_ARGS 2021-05-12T12:49:10.429 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: expr 5 - 1 2021-05-12T12:49:10.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: seq 0 4 2021-05-12T12:49:10.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:10.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2021-05-12T12:49:10.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:10.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:10.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T12:49:10.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:10.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-05-12T12:49:10.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:10.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-05-12T12:49:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-05-12T12:49:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:10.438 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:10.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:10.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:10.439 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:10.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:10.440 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:10.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:10.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:10.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:10.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:10.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:10.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:10.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:10.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:10.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/0 2021-05-12T12:49:10.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:10.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2f662962-3107-40dc-baf1-f0dc7ec6062c 2021-05-12T12:49:10.443 INFO:tasks.workunit.client.0.smithi090.stdout:add osd0 2f662962-3107-40dc-baf1-f0dc7ec6062c 2021-05-12T12:49:10.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 2f662962-3107-40dc-baf1-f0dc7ec6062c' 2021-05-12T12:49:10.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:10.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDGzptgpMiPGhAAX7RASADUS2a3stQXBNc4uw== 2021-05-12T12:49:10.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDGzptgpMiPGhAAX7RASADUS2a3stQXBNc4uw=="}' 2021-05-12T12:49:10.449 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2f662962-3107-40dc-baf1-f0dc7ec6062c -i td/osd-recovery-prio/0/new.json 2021-05-12T12:49:10.768 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:10.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/0/new.json 2021-05-12T12:49:10.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGzptgpMiPGhAAX7RASADUS2a3stQXBNc4uw== --osd-uuid 2f662962-3107-40dc-baf1-f0dc7ec6062c 2021-05-12T12:49:10.803 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.800+0000 7f228f0a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:10.805 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.802+0000 7f228f0a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:10.807 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.803+0000 7f228f0a20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.805+0000 7f228f0a20c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:49:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.805+0000 7f228f0a20c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:49:10.808 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.805+0000 7f228f0a20c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T12:49:10.993 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:10.990+0000 7f228f0a20c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-05-12T12:49:15.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-05-12T12:49:15.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:15.824 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd0 key to auth repository 2021-05-12T12:49:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T12:49:15.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:16.298 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T12:49:16.299 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.0 2021-05-12T12:49:16.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:16.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:16.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:16.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:16.320 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:16.316+0000 7f49ca17f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.323 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:16.318+0000 7f49ca17f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.324 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:16.319+0000 7f49ca17f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:16.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T12:49:16.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:16.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T12:49:16.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:16.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:16.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:16.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:16.660 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:16.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:16.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:49:16.837 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:16.833+0000 7f49ca17f0c0 -1 Falling back to public interface 2021-05-12T12:49:17.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:18.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:18.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:18.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:18.046 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:18.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:18.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:49:18.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:19.067 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:19.064+0000 7f49ca17f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T12:49:19.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:19.410 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:19.410 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:49:19.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:19.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:19.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T12:49:19.859 INFO:tasks.workunit.client.0.smithi090.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/242802,v1:127.0.0.1:6803/242802] [v2:127.0.0.1:6804/242802,v1:127.0.0.1:6805/242802] exists,up 2f662962-3107-40dc-baf1-f0dc7ec6062c 2021-05-12T12:49:19.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:19.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:19.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:19.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:19.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2021-05-12T12:49:19.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:19.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:19.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T12:49:19.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:19.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-05-12T12:49:19.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:19.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:19.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:19.866 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:19.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-05-12T12:49:19.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-05-12T12:49:19.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:19.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:19.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:19.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:19.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:19.868 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:19.868 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:19.869 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:19.869 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:19.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:19.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:19.869 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:19.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:19.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:19.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:19.870 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:19.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:19.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/1 2021-05-12T12:49:19.872 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:19.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=416072e2-14bf-44b5-8f94-fdc37991045f 2021-05-12T12:49:19.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 416072e2-14bf-44b5-8f94-fdc37991045f' 2021-05-12T12:49:19.873 INFO:tasks.workunit.client.0.smithi090.stdout:add osd1 416072e2-14bf-44b5-8f94-fdc37991045f 2021-05-12T12:49:19.873 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:19.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDPzptgHNsNNBAAICFLYOlTkez7bbP2dGQXSg== 2021-05-12T12:49:19.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDPzptgHNsNNBAAICFLYOlTkez7bbP2dGQXSg=="}' 2021-05-12T12:49:19.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 416072e2-14bf-44b5-8f94-fdc37991045f -i td/osd-recovery-prio/1/new.json 2021-05-12T12:49:20.337 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:20.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/1/new.json 2021-05-12T12:49:20.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPzptgHNsNNBAAICFLYOlTkez7bbP2dGQXSg== --osd-uuid 416072e2-14bf-44b5-8f94-fdc37991045f 2021-05-12T12:49:20.372 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.369+0000 7faf016d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:20.374 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.371+0000 7faf016d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:20.375 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.372+0000 7faf016d20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:20.376 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.373+0000 7faf016d20c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:49:20.376 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.373+0000 7faf016d20c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:49:20.377 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.373+0000 7faf016d20c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T12:49:20.448 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:20.446+0000 7faf016d20c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-05-12T12:49:25.365 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-05-12T12:49:25.366 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:25.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T12:49:25.367 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd1 key to auth repository 2021-05-12T12:49:25.367 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:25.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T12:49:25.848 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.1 2021-05-12T12:49:25.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:25.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:25.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:25.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:25.869 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:25.866+0000 7fd596b990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:25.871 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:25.868+0000 7fd596b990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:25.872 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:25.869+0000 7fd596b990c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:26.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T12:49:26.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:26.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T12:49:26.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:26.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:26.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:26.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:26.213 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:26.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:26.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:26.391 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:26.387+0000 7fd596b990c0 -1 Falling back to public interface 2021-05-12T12:49:26.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:27.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:27.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:27.589 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:27.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:27.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:27.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:27.950 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:28.675 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:28.671+0000 7fd596b990c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T12:49:28.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:28.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:28.952 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:49:28.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:28.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:28.953 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:29.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:30.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:30.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:30.358 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:49:30.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:49:30.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:30.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T12:49:30.722 INFO:tasks.workunit.client.0.smithi090.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/244187,v1:127.0.0.1:6811/244187] [v2:127.0.0.1:6812/244187,v1:127.0.0.1:6813/244187] exists,up 416072e2-14bf-44b5-8f94-fdc37991045f 2021-05-12T12:49:30.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:30.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2021-05-12T12:49:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:30.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T12:49:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2021-05-12T12:49:30.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2021-05-12T12:49:30.729 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2021-05-12T12:49:30.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:30.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:30.730 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:30.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:30.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:30.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:30.731 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:30.731 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:30.731 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:30.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:30.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:30.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:30.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:30.732 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:30.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:30.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:30.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:30.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:30.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:30.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:30.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:30.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/2 2021-05-12T12:49:30.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:30.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=24beecef-247f-4c08-acdf-1b0a5bd31ad5 2021-05-12T12:49:30.735 INFO:tasks.workunit.client.0.smithi090.stdout:add osd2 24beecef-247f-4c08-acdf-1b0a5bd31ad5 2021-05-12T12:49:30.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 24beecef-247f-4c08-acdf-1b0a5bd31ad5' 2021-05-12T12:49:30.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDazptgnd/mKxAAS7STFjCcY4myiNrhsPepXg== 2021-05-12T12:49:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDazptgnd/mKxAAS7STFjCcY4myiNrhsPepXg=="}' 2021-05-12T12:49:30.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 24beecef-247f-4c08-acdf-1b0a5bd31ad5 -i td/osd-recovery-prio/2/new.json 2021-05-12T12:49:31.299 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:49:31.310 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/2/new.json 2021-05-12T12:49:31.312 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDazptgnd/mKxAAS7STFjCcY4myiNrhsPepXg== --osd-uuid 24beecef-247f-4c08-acdf-1b0a5bd31ad5 2021-05-12T12:49:31.334 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.331+0000 7fc3e77020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:31.335 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.332+0000 7fc3e77020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:31.336 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.333+0000 7fc3e77020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.334+0000 7fc3e77020c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.334+0000 7fc3e77020c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.334+0000 7fc3e77020c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-05-12T12:49:31.382 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:31.379+0000 7fc3e77020c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2021-05-12T12:49:36.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2021-05-12T12:49:36.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:36.281 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd2 key to auth repository 2021-05-12T12:49:36.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T12:49:36.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:36.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T12:49:36.763 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.2 2021-05-12T12:49:36.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:36.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:36.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:36.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:36.784 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:36.780+0000 7f40b31e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:36.785 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:36.782+0000 7f40b31e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:36.786 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:36.783+0000 7f40b31e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:37.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T12:49:37.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:37.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T12:49:37.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:37.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:37.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:37.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:37.121 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:37.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:37.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:37.303 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:37.298+0000 7f40b31e60c0 -1 Falling back to public interface 2021-05-12T12:49:37.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:38.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:38.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:38.475 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:38.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:38.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:38.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:38.837 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:39.480 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:39.476+0000 7f40b31e60c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T12:49:39.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:39.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:39.839 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:49:39.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:39.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:39.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T12:49:40.250 INFO:tasks.workunit.client.0.smithi090.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/245598,v1:127.0.0.1:6819/245598] [v2:127.0.0.1:6820/245598,v1:127.0.0.1:6821/245598] exists,up 24beecef-247f-4c08-acdf-1b0a5bd31ad5 2021-05-12T12:49:40.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:40.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:40.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:40.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:40.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2021-05-12T12:49:40.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:40.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:40.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T12:49:40.256 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:40.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2021-05-12T12:49:40.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:40.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:40.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:40.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:40.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2021-05-12T12:49:40.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2021-05-12T12:49:40.258 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:40.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:40.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:40.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:40.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:40.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:40.260 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:40.260 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:40.260 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:40.260 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:40.261 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:40.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:40.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:40.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:40.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:40.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:40.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:40.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:40.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/3 2021-05-12T12:49:40.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:40.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4a27835d-c6b0-4ed4-9441-747dc1778bef 2021-05-12T12:49:40.264 INFO:tasks.workunit.client.0.smithi090.stdout:add osd3 4a27835d-c6b0-4ed4-9441-747dc1778bef 2021-05-12T12:49:40.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 4a27835d-c6b0-4ed4-9441-747dc1778bef' 2021-05-12T12:49:40.265 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:40.267 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDkzptgrn7EDxAA4uBReJ/bt0xWNjf+YptdHw== 2021-05-12T12:49:40.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDkzptgrn7EDxAA4uBReJ/bt0xWNjf+YptdHw=="}' 2021-05-12T12:49:40.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4a27835d-c6b0-4ed4-9441-747dc1778bef -i td/osd-recovery-prio/3/new.json 2021-05-12T12:49:40.722 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:49:40.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/3/new.json 2021-05-12T12:49:40.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDkzptgrn7EDxAA4uBReJ/bt0xWNjf+YptdHw== --osd-uuid 4a27835d-c6b0-4ed4-9441-747dc1778bef 2021-05-12T12:49:40.757 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.753+0000 7f835a5050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:40.759 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.755+0000 7f835a5050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:40.761 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.756+0000 7f835a5050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:40.761 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.757+0000 7f835a5050c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:40.761 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.757+0000 7f835a5050c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:40.762 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.757+0000 7f835a5050c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-05-12T12:49:40.849 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:40.845+0000 7f835a5050c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2021-05-12T12:49:45.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2021-05-12T12:49:45.924 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:45.925 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd3 key to auth repository 2021-05-12T12:49:45.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T12:49:45.926 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:46.407 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T12:49:46.408 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.3 2021-05-12T12:49:46.408 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:46.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:46.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:46.409 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:46.428 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:46.424+0000 7f980388d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:46.430 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:46.426+0000 7f980388d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:46.432 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:46.427+0000 7f980388d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:46.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T12:49:46.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:46.770 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T12:49:46.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:46.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:46.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:46.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:46.772 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:46.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:46.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:46.948 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:46.944+0000 7f980388d0c0 -1 Falling back to public interface 2021-05-12T12:49:47.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:48.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:48.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:48.159 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:48.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:48.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:48.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:48.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:49.213 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:49.209+0000 7f980388d0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T12:49:49.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:49.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:49.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:49:49.538 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:49:49.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:49.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:49.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:50.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:50.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:50.907 INFO:tasks.workunit.client.0.smithi090.stdout:3 2021-05-12T12:49:50.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T12:49:50.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:50.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T12:49:51.270 INFO:tasks.workunit.client.0.smithi090.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/246983,v1:127.0.0.1:6827/246983] [v2:127.0.0.1:6828/246983,v1:127.0.0.1:6829/246983] exists,up 4a27835d-c6b0-4ed4-9441-747dc1778bef 2021-05-12T12:49:51.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:49:51.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:49:51.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:49:51.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T12:49:51.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2021-05-12T12:49:51.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T12:49:51.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T12:49:51.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-05-12T12:49:51.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T12:49:51.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2021-05-12T12:49:51.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T12:49:51.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T12:49:51.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T12:49:51.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T12:49:51.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2021-05-12T12:49:51.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2021-05-12T12:49:51.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T12:49:51.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T12:49:51.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T12:49:51.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T12:49:51.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T12:49:51.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T12:49:51.279 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T12:49:51.280 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:49:51.280 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:49:51.280 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:51.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' 2021-05-12T12:49:51.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T12:49:51.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T12:49:51.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T12:49:51.281 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T12:49:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T12:49:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T12:49:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T12:49:51.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T12:49:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T12:49:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/4 2021-05-12T12:49:51.283 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T12:49:51.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27 2021-05-12T12:49:51.284 INFO:tasks.workunit.client.0.smithi090.stdout:add osd4 60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27 2021-05-12T12:49:51.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27' 2021-05-12T12:49:51.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T12:49:51.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDvzptgt+/yEBAASlx8rA8rYy/iP7jUyvgOLg== 2021-05-12T12:49:51.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDvzptgt+/yEBAASlx8rA8rYy/iP7jUyvgOLg=="}' 2021-05-12T12:49:51.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27 -i td/osd-recovery-prio/4/new.json 2021-05-12T12:49:51.742 INFO:tasks.workunit.client.0.smithi090.stdout:4 2021-05-12T12:49:51.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/4/new.json 2021-05-12T12:49:51.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDvzptgt+/yEBAASlx8rA8rYy/iP7jUyvgOLg== --osd-uuid 60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27 2021-05-12T12:49:51.777 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.773+0000 7ff85a0e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:51.779 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.775+0000 7ff85a0e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:51.781 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.776+0000 7ff85a0e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:51.782 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.777+0000 7ff85a0e60c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:51.782 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.777+0000 7ff85a0e60c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:51.782 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.777+0000 7ff85a0e60c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-05-12T12:49:51.911 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:51.907+0000 7ff85a0e60c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2021-05-12T12:49:56.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2021-05-12T12:49:56.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T12:49:56.768 INFO:tasks.workunit.client.0.smithi090.stdout:adding osd4 key to auth repository 2021-05-12T12:49:56.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2021-05-12T12:49:56.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T12:49:57.301 INFO:tasks.workunit.client.0.smithi090.stdout:start osd.4 2021-05-12T12:49:57.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2021-05-12T12:49:57.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=d60016b5-944b-4ce0-8b90-1a87e3df6db4 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185878/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T12:49:57.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T12:49:57.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T12:49:57.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T12:49:57.389 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:57.318+0000 7f8219c440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:57.390 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:57.320+0000 7f8219c440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:57.390 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:57.322+0000 7f8219c440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T12:49:57.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2021-05-12T12:49:57.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T12:49:57.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=4 2021-05-12T12:49:57.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T12:49:57.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T12:49:57.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:57.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T12:49:57.663 INFO:tasks.workunit.client.0.smithi090.stdout:0 2021-05-12T12:49:57.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:57.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:57.841 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:49:57.838+0000 7f8219c440c0 -1 Falling back to public interface 2021-05-12T12:49:58.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:49:59.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:49:59.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:49:59.017 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T12:49:59.017 INFO:tasks.workunit.client.0.smithi090.stdout:1 2021-05-12T12:49:59.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:49:59.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:49:59.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T12:50:00.122 INFO:tasks.workunit.client.0.smithi090.stderr:2021-05-12T12:50:00.118+0000 7f8219c440c0 -1 osd.4 0 log_to_monitors {default=true} 2021-05-12T12:50:00.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T12:50:00.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T12:50:00.381 INFO:tasks.workunit.client.0.smithi090.stdout:2 2021-05-12T12:50:00.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T12:50:00.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T12:50:00.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.4 up' 2021-05-12T12:50:00.823 INFO:tasks.workunit.client.0.smithi090.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/248402,v1:127.0.0.1:6835/248402] [v2:127.0.0.1:6836/248402,v1:127.0.0.1:6837/248402] exists,up 60dc5fbc-f289-43f4-a5ed-1eb8e1ccdf27 2021-05-12T12:50:00.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T12:50:00.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T12:50:00.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T12:50:00.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: seq 1 10 2021-05-12T12:50:00.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:00.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test1 1 1 2021-05-12T12:50:00.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-05-12T12:50:01.593 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test1' already exists 2021-05-12T12:50:01.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:02.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test1 size 2 2021-05-12T12:50:03.374 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 1 size to 2 2021-05-12T12:50:03.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:03.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test2 1 1 2021-05-12T12:50:03.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-05-12T12:50:03.975 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test2' already exists 2021-05-12T12:50:03.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:04.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test2 size 2 2021-05-12T12:50:05.638 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 2 size to 2 2021-05-12T12:50:05.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:05.647 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test3 1 1 2021-05-12T12:50:05.648 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-05-12T12:50:06.521 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test3' already exists 2021-05-12T12:50:06.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:07.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-05-12T12:50:08.382 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 3 size to 2 2021-05-12T12:50:08.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:08.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test4 1 1 2021-05-12T12:50:08.391 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2021-05-12T12:50:08.948 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test4' already exists 2021-05-12T12:50:08.957 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:09.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test4 size 2 2021-05-12T12:50:10.739 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 4 size to 2 2021-05-12T12:50:10.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:10.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test5 1 1 2021-05-12T12:50:10.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2021-05-12T12:50:11.467 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test5' already exists 2021-05-12T12:50:11.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:12.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test5 size 2 2021-05-12T12:50:13.613 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 5 size to 2 2021-05-12T12:50:13.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:13.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test6 1 1 2021-05-12T12:50:13.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2021-05-12T12:50:14.182 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test6' already exists 2021-05-12T12:50:14.191 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:15.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test6 size 2 2021-05-12T12:50:15.919 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 6 size to 2 2021-05-12T12:50:15.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:15.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test7 1 1 2021-05-12T12:50:15.928 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2021-05-12T12:50:16.518 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test7' already exists 2021-05-12T12:50:16.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:17.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-05-12T12:50:18.467 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 7 size to 2 2021-05-12T12:50:18.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:18.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test8 1 1 2021-05-12T12:50:18.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2021-05-12T12:50:19.104 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test8' already exists 2021-05-12T12:50:19.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:20.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test8 size 2 2021-05-12T12:50:20.899 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 8 size to 2 2021-05-12T12:50:20.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:20.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test9 1 1 2021-05-12T12:50:20.909 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2021-05-12T12:50:21.653 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test9' already exists 2021-05-12T12:50:21.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:22.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test9 size 2 2021-05-12T12:50:23.534 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 9 size to 2 2021-05-12T12:50:23.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:23.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test10 1 1 2021-05-12T12:50:23.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2021-05-12T12:50:24.198 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test10' already exists 2021-05-12T12:50:24.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T12:50:25.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test10 size 2 2021-05-12T12:50:26.055 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 10 size to 2 2021-05-12T12:50:26.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: sleep 5 2021-05-12T12:50:31.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:65: TEST_recovery_priority: wait_for_clean 2021-05-12T12:50:31.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:50:31.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:50:31.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:50:31.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:50:31.067 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:50:31.068 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:50:31.068 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:50:31.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:50:31.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:50:31.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:50:31.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:50:31.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:50:31.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:50:31.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:50:31.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:50:31.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:50:31.525 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:50:31.525 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:50:31.525 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:50:31.525 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:50:31.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:50:31.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:31.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:50:31.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836497 2021-05-12T12:50:31.651 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836497 2021-05-12T12:50:31.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497' 2021-05-12T12:50:31.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:31.652 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:50:31.777 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705679 2021-05-12T12:50:31.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705679 2021-05-12T12:50:31.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705679' 2021-05-12T12:50:31.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:31.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:50:31.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542158 2021-05-12T12:50:31.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542158 2021-05-12T12:50:31.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705679 2-60129542158' 2021-05-12T12:50:31.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:31.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:50:32.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411339 2021-05-12T12:50:32.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411339 2021-05-12T12:50:32.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705679 2-60129542158 3-77309411339' 2021-05-12T12:50:32.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:32.031 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:50:32.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247817 2021-05-12T12:50:32.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247817 2021-05-12T12:50:32.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836497 1-38654705679 2-60129542158 3-77309411339 4-98784247817' 2021-05-12T12:50:32.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:32.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2021-05-12T12:50:32.159 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:32.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:50:32.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836497 2021-05-12T12:50:32.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:32.164 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836497 2021-05-12T12:50:32.165 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-05-12T12:50:32.165 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836497 2021-05-12T12:50:32.166 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:32.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836495 -lt 21474836497 2021-05-12T12:50:32.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:50:33.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:50:33.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836497 -lt 21474836497 2021-05-12T12:50:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:33.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705679 2021-05-12T12:50:33.880 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:33.881 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:50:33.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705679 2021-05-12T12:50:33.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:33.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705679 2021-05-12T12:50:33.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2021-05-12T12:50:33.883 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705679 2021-05-12T12:50:33.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:50:34.238 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705679 -lt 38654705679 2021-05-12T12:50:34.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:34.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542158 2021-05-12T12:50:34.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:34.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:50:34.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542158 2021-05-12T12:50:34.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:34.241 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542158 2021-05-12T12:50:34.242 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542158' 2021-05-12T12:50:34.242 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542158 2021-05-12T12:50:34.242 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:50:34.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542158 -lt 60129542158 2021-05-12T12:50:34.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:34.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411339 2021-05-12T12:50:34.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:34.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:50:34.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411339 2021-05-12T12:50:34.622 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411339 2021-05-12T12:50:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411339' 2021-05-12T12:50:34.624 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411339 2021-05-12T12:50:34.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:50:34.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411339 -lt 77309411339 2021-05-12T12:50:34.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:34.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247817 2021-05-12T12:50:34.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:34.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:50:34.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247817 2021-05-12T12:50:34.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:34.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247817 2021-05-12T12:50:34.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247817' 2021-05-12T12:50:34.984 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247817 2021-05-12T12:50:34.985 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:50:35.342 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247817 -lt 98784247817 2021-05-12T12:50:35.342 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:50:35.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:35.343 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:35.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 10 == 0 2021-05-12T12:50:35.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:50:35.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:50:35.812 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:50:35.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:50:35.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:50:35.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:50:35.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:50:36.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=10 2021-05-12T12:50:36.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:50:36.172 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:36.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:36.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 10 = 10 2021-05-12T12:50:36.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:50:36.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:50:36.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:67: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:50:36.971 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:50:36.972 INFO:tasks.workunit.client.0.smithi090.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:26.301339+0000 0'0 78:19 [2,1] 2 [2,1] 2 0'0 2021-05-12T12:50:23.928014+0000 0'0 2021-05-12T12:50:23.928014+0000 0 2021-05-12T12:50:36.972 INFO:tasks.workunit.client.0.smithi090.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:24.514512+0000 0'0 77:24 [1,2] 1 [1,2] 1 0'0 2021-05-12T12:50:21.378841+0000 0'0 2021-05-12T12:50:21.378841+0000 0 2021-05-12T12:50:36.972 INFO:tasks.workunit.client.0.smithi090.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:21.861545+0000 0'0 77:27 [3,4] 3 [3,4] 3 0'0 2021-05-12T12:50:18.898742+0000 0'0 2021-05-12T12:50:18.898742+0000 0 2021-05-12T12:50:36.972 INFO:tasks.workunit.client.0.smithi090.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:19.125381+0000 0'0 77:34 [1,4] 1 [1,4] 1 0'0 2021-05-12T12:50:16.255616+0000 0'0 2021-05-12T12:50:16.255616+0000 0 2021-05-12T12:50:36.973 INFO:tasks.workunit.client.0.smithi090.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:06.329821+0000 0'0 77:49 [3,1] 3 [3,1] 3 0'0 2021-05-12T12:50:03.653734+0000 0'0 2021-05-12T12:50:03.653734+0000 0 2021-05-12T12:50:36.973 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:04.115463+0000 0'0 77:58 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:50:01.133820+0000 0'0 2021-05-12T12:50:01.133820+0000 0 2021-05-12T12:50:36.973 INFO:tasks.workunit.client.0.smithi090.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:09.046666+0000 0'0 77:51 [1,2] 1 [1,2] 1 0'0 2021-05-12T12:50:06.202217+0000 0'0 2021-05-12T12:50:06.202217+0000 0 2021-05-12T12:50:36.973 INFO:tasks.workunit.client.0.smithi090.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:12.539153+0000 0'0 77:42 [4,0] 4 [4,0] 4 0'0 2021-05-12T12:50:08.757683+0000 0'0 2021-05-12T12:50:08.757683+0000 0 2021-05-12T12:50:36.974 INFO:tasks.workunit.client.0.smithi090.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:14.421284+0000 0'0 77:40 [3,4] 3 [3,4] 3 0'0 2021-05-12T12:50:11.092303+0000 0'0 2021-05-12T12:50:11.092303+0000 0 2021-05-12T12:50:36.974 INFO:tasks.workunit.client.0.smithi090.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T12:50:17.540133+0000 0'0 77:34 [4,2] 4 [4,2] 4 0'0 2021-05-12T12:50:13.942036+0000 0'0 2021-05-12T12:50:13.942036+0000 0 2021-05-12T12:50:36.974 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:50:36.974 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:50:36.975 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:50:36.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:71: TEST_recovery_priority: local PG1 2021-05-12T12:50:36.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:72: TEST_recovery_priority: local POOLNUM1 2021-05-12T12:50:36.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:73: TEST_recovery_priority: local pool1 2021-05-12T12:50:36.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local chk_osd1_1 2021-05-12T12:50:36.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local chk_osd1_2 2021-05-12T12:50:36.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local PG2 2021-05-12T12:50:36.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local POOLNUM2 2021-05-12T12:50:36.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:79: TEST_recovery_priority: local pool2 2021-05-12T12:50:36.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local chk_osd2 2021-05-12T12:50:36.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local PG3 2021-05-12T12:50:36.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local POOLNUM3 2021-05-12T12:50:36.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:84: TEST_recovery_priority: local pool3 2021-05-12T12:50:36.987 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: seq 1 10 2021-05-12T12:50:36.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:36.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 1.0 --format=json 2021-05-12T12:50:36.987 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:37.333 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:37.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:37.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:37.336 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-05-12T12:50:37.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z '' ']' 2021-05-12T12:50:37.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: PG1=1.0 2021-05-12T12:50:37.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: POOLNUM1=1 2021-05-12T12:50:37.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:95: TEST_recovery_priority: pool1=test1 2021-05-12T12:50:37.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: chk_osd1_1=1 2021-05-12T12:50:37.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: chk_osd1_2=0 2021-05-12T12:50:37.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:37.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 2.0 --format=json 2021-05-12T12:50:37.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:37.684 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:37.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-05-12T12:50:37.685 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:37.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=1 2021-05-12T12:50:37.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:37.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-05-12T12:50:37.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-05-12T12:50:37.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:37.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 3.0 --format=json 2021-05-12T12:50:37.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:38.036 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:38.037 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-05-12T12:50:38.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:38.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-05-12T12:50:38.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: PG2=3.0 2021-05-12T12:50:38.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: POOLNUM2=3 2021-05-12T12:50:38.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:102: TEST_recovery_priority: pool2=test3 2021-05-12T12:50:38.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: chk_osd2=2 2021-05-12T12:50:38.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:38.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 4.0 --format=json 2021-05-12T12:50:38.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:38.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-05-12T12:50:38.392 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-05-12T12:50:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-05-12T12:50:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-05-12T12:50:38.394 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 5.0 --format=json 2021-05-12T12:50:38.395 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:38.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-05-12T12:50:38.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:38.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-05-12T12:50:38.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:38.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-05-12T12:50:38.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-05-12T12:50:38.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:38.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 6.0 --format=json 2021-05-12T12:50:38.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:39.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:39.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-05-12T12:50:39.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:39.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-05-12T12:50:39.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:39.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-05-12T12:50:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-05-12T12:50:39.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 7.0 --format=json 2021-05-12T12:50:39.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-05-12T12:50:39.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T12:50:39.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-05-12T12:50:39.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T12:50:39.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-05-12T12:50:39.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-05-12T12:50:39.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-05-12T12:50:39.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-05-12T12:50:39.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: PG3=7.0 2021-05-12T12:50:39.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: POOLNUM3=7 2021-05-12T12:50:39.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:108: TEST_recovery_priority: pool3=test7 2021-05-12T12:50:39.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: break 2021-05-12T12:50:39.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2021-05-12T12:50:39.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:114: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2021-05-12T12:50:39.474 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: seq 1 10 2021-05-12T12:50:39.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:39.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-05-12T12:50:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-05-12T12:50:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test2 2021-05-12T12:50:39.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-05-12T12:50:39.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-05-12T12:50:39.990 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test2' does not exist 2021-05-12T12:50:39.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:40.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-05-12T12:50:40.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:40.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-05-12T12:50:40.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test4 2021-05-12T12:50:40.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2021-05-12T12:50:40.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-05-12T12:50:40.544 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test4' does not exist 2021-05-12T12:50:40.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:40.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-05-12T12:50:40.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test5 2021-05-12T12:50:40.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2021-05-12T12:50:40.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-05-12T12:50:41.075 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test5' does not exist 2021-05-12T12:50:41.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:41.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-05-12T12:50:41.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test6 2021-05-12T12:50:41.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2021-05-12T12:50:41.086 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-05-12T12:50:41.754 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test6' does not exist 2021-05-12T12:50:41.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:41.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-05-12T12:50:41.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:41.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-05-12T12:50:41.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test8 2021-05-12T12:50:41.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2021-05-12T12:50:41.765 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-05-12T12:50:42.421 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test8' does not exist 2021-05-12T12:50:42.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:42.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-05-12T12:50:42.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test9 2021-05-12T12:50:42.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2021-05-12T12:50:42.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-05-12T12:50:43.048 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test9' does not exist 2021-05-12T12:50:43.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-05-12T12:50:43.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-05-12T12:50:43.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test10 2021-05-12T12:50:43.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2021-05-12T12:50:43.058 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-05-12T12:50:43.851 INFO:tasks.workunit.client.0.smithi090.stderr:pool 'test10' does not exist 2021-05-12T12:50:43.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:128: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-05-12T12:50:44.590 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 3 size to 1 2021-05-12T12:50:44.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:129: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-05-12T12:50:45.486 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 7 size to 1 2021-05-12T12:50:45.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:130: TEST_recovery_priority: wait_for_clean 2021-05-12T12:50:45.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T12:50:45.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:50:45.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:50:45.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:50:45.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:50:45.499 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:50:45.499 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:50:45.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:50:45.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:50:45.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:50:45.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:50:45.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:50:45.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:50:45.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:50:45.587 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:50:46.024 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:50:46.025 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:50:46.025 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:50:46.025 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:50:46.026 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:50:46.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:50:46.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:46.026 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:50:46.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836502 2021-05-12T12:50:46.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836502 2021-05-12T12:50:46.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836502' 2021-05-12T12:50:46.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:46.153 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:50:46.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705684 2021-05-12T12:50:46.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705684 2021-05-12T12:50:46.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836502 1-38654705684' 2021-05-12T12:50:46.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:46.279 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:50:46.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542163 2021-05-12T12:50:46.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542163 2021-05-12T12:50:46.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836502 1-38654705684 2-60129542163' 2021-05-12T12:50:46.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:46.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:50:46.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411344 2021-05-12T12:50:46.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411344 2021-05-12T12:50:46.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836502 1-38654705684 2-60129542163 3-77309411344' 2021-05-12T12:50:46.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:50:46.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:50:46.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247822 2021-05-12T12:50:46.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247822 2021-05-12T12:50:46.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836502 1-38654705684 2-60129542163 3-77309411344 4-98784247822' 2021-05-12T12:50:46.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:46.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2021-05-12T12:50:46.659 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:46.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:50:46.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836502 2021-05-12T12:50:46.660 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:46.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836502 2021-05-12T12:50:46.661 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-05-12T12:50:46.661 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836502 2021-05-12T12:50:46.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:47.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836499 -lt 21474836502 2021-05-12T12:50:47.015 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:50:48.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:50:48.017 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:50:48.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836502 -lt 21474836502 2021-05-12T12:50:48.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:48.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705684 2021-05-12T12:50:48.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:48.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:50:48.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705684 2021-05-12T12:50:48.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:48.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705684 2021-05-12T12:50:48.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705684' 2021-05-12T12:50:48.384 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705684 2021-05-12T12:50:48.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:50:48.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705684 -lt 38654705684 2021-05-12T12:50:48.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:48.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542163 2021-05-12T12:50:48.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:48.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:50:48.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542163 2021-05-12T12:50:48.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:48.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542163 2021-05-12T12:50:48.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2021-05-12T12:50:48.748 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542163 2021-05-12T12:50:48.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:50:49.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542163 -lt 60129542163 2021-05-12T12:50:49.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:49.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411344 2021-05-12T12:50:49.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:49.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:50:49.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411344 2021-05-12T12:50:49.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:49.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411344 2021-05-12T12:50:49.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411344' 2021-05-12T12:50:49.133 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411344 2021-05-12T12:50:49.133 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:50:49.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411344 -lt 77309411344 2021-05-12T12:50:49.499 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:50:49.499 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247822 2021-05-12T12:50:49.500 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:50:49.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:50:49.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247822 2021-05-12T12:50:49.501 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:50:49.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247822 2021-05-12T12:50:49.502 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247822' 2021-05-12T12:50:49.502 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247822 2021-05-12T12:50:49.502 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:50:49.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247822 -lt 98784247822 2021-05-12T12:50:49.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:50:49.861 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:49.861 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:50.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:50:50.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:50:50.338 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:50:50.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:50:50.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:50:50.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:50:50.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:50:50.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:50:50.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=3 2021-05-12T12:50:50.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:50:50.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:50:50.682 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:50:51.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 = 3 2021-05-12T12:50:51.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T12:50:51.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T12:50:51.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-05-12T12:50:51.198 INFO:tasks.workunit.client.0.smithi090.stderr:10+0 records in 2021-05-12T12:50:51.198 INFO:tasks.workunit.client.0.smithi090.stderr:10+0 records out 2021-05-12T12:50:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0462453 s, 227 MB/s 2021-05-12T12:50:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: p=1 2021-05-12T12:50:51.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:50:51.199 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:50:51.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:51.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-05-12T12:50:51.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:51.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-05-12T12:50:52.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:52.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-05-12T12:50:53.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:53.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-05-12T12:50:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:54.220 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-05-12T12:50:54.797 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:54.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-05-12T12:50:56.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:56.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-05-12T12:50:56.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:56.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-05-12T12:50:57.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:57.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-05-12T12:50:58.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:58.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-05-12T12:50:59.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:50:59.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-05-12T12:51:00.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:00.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-05-12T12:51:00.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:00.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-05-12T12:51:01.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:01.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-05-12T12:51:02.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:02.776 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-05-12T12:51:03.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:03.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-05-12T12:51:04.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:04.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-05-12T12:51:05.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:05.424 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-05-12T12:51:06.035 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:06.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-05-12T12:51:06.638 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:06.639 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-05-12T12:51:07.374 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:07.375 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-05-12T12:51:08.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:08.786 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-05-12T12:51:09.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:09.406 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-05-12T12:51:09.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:09.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-05-12T12:51:12.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:12.607 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-05-12T12:51:13.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:13.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-05-12T12:51:14.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:14.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-05-12T12:51:15.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:15.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-05-12T12:51:16.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:16.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-05-12T12:51:17.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:17.170 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-05-12T12:51:17.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:17.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-05-12T12:51:18.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:18.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-05-12T12:51:19.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:19.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-05-12T12:51:20.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:20.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-05-12T12:51:21.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:21.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-05-12T12:51:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:21.956 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-05-12T12:51:22.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:22.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-05-12T12:51:24.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:24.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-05-12T12:51:24.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:24.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-05-12T12:51:25.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:25.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-05-12T12:51:26.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:26.897 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-05-12T12:51:27.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:27.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-05-12T12:51:28.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:28.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-05-12T12:51:29.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:29.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-05-12T12:51:30.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:30.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-05-12T12:51:30.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:30.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-05-12T12:51:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:31.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-05-12T12:51:33.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:33.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-05-12T12:51:34.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:34.665 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-05-12T12:51:35.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:35.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-05-12T12:51:36.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:36.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-05-12T12:51:36.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:36.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-05-12T12:51:37.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:37.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-05-12T12:51:38.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:38.929 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-05-12T12:51:39.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:39.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-05-12T12:51:40.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:40.126 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-05-12T12:51:41.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:41.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-05-12T12:51:42.348 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:42.349 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-05-12T12:51:42.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:42.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-05-12T12:51:43.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:43.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-05-12T12:51:44.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:44.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-05-12T12:51:45.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:45.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-05-12T12:51:46.171 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:46.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-05-12T12:51:47.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:47.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-05-12T12:51:48.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:48.243 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-05-12T12:51:48.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:48.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-05-12T12:51:49.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:49.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-05-12T12:51:50.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:50.910 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-05-12T12:51:51.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:51.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-05-12T12:51:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:52.150 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-05-12T12:51:52.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:52.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-05-12T12:51:55.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:55.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-05-12T12:51:56.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:56.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-05-12T12:51:57.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:57.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-05-12T12:51:57.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:57.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-05-12T12:51:58.344 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:58.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-05-12T12:51:59.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:51:59.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-05-12T12:52:00.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:00.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-05-12T12:52:01.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:01.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-05-12T12:52:01.697 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:01.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-05-12T12:52:03.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:03.050 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-05-12T12:52:03.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:03.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-05-12T12:52:04.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:04.331 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-05-12T12:52:05.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:05.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-05-12T12:52:06.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:06.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-05-12T12:52:07.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:07.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-05-12T12:52:07.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:07.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-05-12T12:52:09.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:09.079 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-05-12T12:52:09.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:09.773 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-05-12T12:52:10.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:10.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-05-12T12:52:11.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:11.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-05-12T12:52:12.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:12.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-05-12T12:52:13.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:13.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-05-12T12:52:13.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:13.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-05-12T12:52:16.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:16.362 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-05-12T12:52:18.675 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:18.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-05-12T12:52:19.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:19.345 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-05-12T12:52:20.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:20.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-05-12T12:52:21.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:21.361 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-05-12T12:52:21.969 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:21.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-05-12T12:52:22.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:22.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-05-12T12:52:24.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:24.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-05-12T12:52:24.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:24.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-05-12T12:52:26.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:26.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-05-12T12:52:27.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:27.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-05-12T12:52:28.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:28.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-05-12T12:52:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:28.939 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-05-12T12:52:30.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:30.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-05-12T12:52:30.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:30.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-05-12T12:52:31.523 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:31.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-05-12T12:52:32.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:32.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-05-12T12:52:33.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:33.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-05-12T12:52:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-05-12T12:52:34.692 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:34.693 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-05-12T12:52:36.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:36.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-05-12T12:52:36.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:36.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-05-12T12:52:37.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:37.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-05-12T12:52:37.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:37.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-05-12T12:52:40.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:40.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-05-12T12:52:41.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:41.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-05-12T12:52:42.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:42.350 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-05-12T12:52:42.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:42.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-05-12T12:52:43.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:43.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-05-12T12:52:44.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:44.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-05-12T12:52:45.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:45.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-05-12T12:52:46.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:46.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-05-12T12:52:46.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:46.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-05-12T12:52:48.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:48.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-05-12T12:52:49.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:49.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-05-12T12:52:49.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:49.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-05-12T12:52:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:50.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-05-12T12:52:51.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:51.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-05-12T12:52:52.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:52.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-05-12T12:52:53.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:53.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-05-12T12:52:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:54.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-05-12T12:52:55.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:55.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-05-12T12:52:55.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:55.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-05-12T12:52:56.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:56.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-05-12T12:52:57.784 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:57.785 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-05-12T12:52:58.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:58.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-05-12T12:52:59.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:52:59.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-05-12T12:53:01.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:01.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-05-12T12:53:02.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:02.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-05-12T12:53:03.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:03.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-05-12T12:53:04.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:04.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-05-12T12:53:04.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:04.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-05-12T12:53:05.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:05.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-05-12T12:53:06.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:06.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-05-12T12:53:07.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:07.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-05-12T12:53:07.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:07.819 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-05-12T12:53:08.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:08.472 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-05-12T12:53:10.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:10.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-05-12T12:53:10.970 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:10.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-05-12T12:53:12.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:12.415 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-05-12T12:53:13.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:13.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-05-12T12:53:14.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:14.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-05-12T12:53:15.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:15.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-05-12T12:53:16.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:16.668 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-05-12T12:53:17.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:17.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-05-12T12:53:18.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:18.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-05-12T12:53:19.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:19.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-05-12T12:53:20.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:20.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-05-12T12:53:21.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:21.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-05-12T12:53:22.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-05-12T12:53:22.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:22.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-05-12T12:53:25.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:25.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-05-12T12:53:25.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:25.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-05-12T12:53:26.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:26.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-05-12T12:53:27.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:27.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-05-12T12:53:28.447 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:28.448 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-05-12T12:53:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:29.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-05-12T12:53:30.579 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:30.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-05-12T12:53:31.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:31.190 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-05-12T12:53:31.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:31.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-05-12T12:53:32.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:32.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-05-12T12:53:33.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:33.787 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-05-12T12:53:34.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:34.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-05-12T12:53:34.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:34.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-05-12T12:53:35.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:35.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-05-12T12:53:36.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:36.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-05-12T12:53:37.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:37.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-05-12T12:53:38.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:38.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-05-12T12:53:38.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:38.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-05-12T12:53:40.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:40.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-05-12T12:53:40.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:40.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-05-12T12:53:41.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:41.450 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-05-12T12:53:42.811 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:42.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-05-12T12:53:43.371 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:43.372 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-05-12T12:53:44.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:44.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-05-12T12:53:46.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:46.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-05-12T12:53:47.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:47.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-05-12T12:53:47.936 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:47.937 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-05-12T12:53:49.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:49.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-05-12T12:53:49.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:49.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-05-12T12:53:50.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:50.705 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-05-12T12:53:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:52.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-05-12T12:53:52.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:52.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-05-12T12:53:53.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:53.304 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-05-12T12:53:53.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:53.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-05-12T12:53:55.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:55.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-05-12T12:53:56.055 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 1 + 1 2021-05-12T12:53:56.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=2 2021-05-12T12:53:56.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:53:56.056 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:53:56.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2021-05-12T12:53:56.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.381 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2021-05-12T12:53:56.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:56.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2021-05-12T12:53:57.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:57.016 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2021-05-12T12:53:58.259 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:58.260 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2021-05-12T12:53:58.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:58.873 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2021-05-12T12:53:59.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:53:59.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2021-05-12T12:54:00.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:00.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2021-05-12T12:54:00.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:00.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2021-05-12T12:54:01.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:01.189 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2021-05-12T12:54:01.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:01.559 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2021-05-12T12:54:01.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:01.917 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2021-05-12T12:54:02.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:02.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2021-05-12T12:54:03.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:03.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2021-05-12T12:54:03.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:03.373 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.425 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2021-05-12T12:54:04.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:04.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2021-05-12T12:54:05.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:05.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2021-05-12T12:54:05.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:05.453 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2021-05-12T12:54:05.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:05.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2021-05-12T12:54:06.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:06.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2021-05-12T12:54:06.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:06.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2021-05-12T12:54:07.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:07.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2021-05-12T12:54:07.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:07.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2021-05-12T12:54:08.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:08.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2021-05-12T12:54:09.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:09.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2021-05-12T12:54:09.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:09.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2021-05-12T12:54:10.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:10.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2021-05-12T12:54:10.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:10.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2021-05-12T12:54:11.151 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:11.152 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2021-05-12T12:54:11.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:11.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2021-05-12T12:54:11.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:11.857 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2021-05-12T12:54:12.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:12.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2021-05-12T12:54:13.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:13.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2021-05-12T12:54:14.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:14.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2021-05-12T12:54:14.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:14.807 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2021-05-12T12:54:15.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:15.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2021-05-12T12:54:15.519 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:15.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2021-05-12T12:54:15.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:15.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2021-05-12T12:54:16.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:16.806 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2021-05-12T12:54:17.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:17.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2021-05-12T12:54:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:18.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2021-05-12T12:54:18.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:18.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2021-05-12T12:54:18.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:18.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2021-05-12T12:54:19.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:19.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.601 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2021-05-12T12:54:20.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:20.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2021-05-12T12:54:21.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:21.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2021-05-12T12:54:21.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:21.948 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2021-05-12T12:54:22.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:22.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2021-05-12T12:54:23.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:23.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2021-05-12T12:54:23.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:23.562 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2021-05-12T12:54:23.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:23.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2021-05-12T12:54:24.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:24.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2021-05-12T12:54:25.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:25.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2021-05-12T12:54:26.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:26.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2021-05-12T12:54:27.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:27.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2021-05-12T12:54:27.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:27.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2021-05-12T12:54:27.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:27.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2021-05-12T12:54:28.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:28.159 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.194 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2021-05-12T12:54:29.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:29.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.291 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2021-05-12T12:54:30.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:30.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2021-05-12T12:54:31.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:31.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2021-05-12T12:54:32.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:32.414 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2021-05-12T12:54:32.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:32.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2021-05-12T12:54:33.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:33.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2021-05-12T12:54:33.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:33.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2021-05-12T12:54:33.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:33.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2021-05-12T12:54:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2021-05-12T12:54:34.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:34.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2021-05-12T12:54:35.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:35.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2021-05-12T12:54:36.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:36.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2021-05-12T12:54:36.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:36.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2021-05-12T12:54:37.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:37.213 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2021-05-12T12:54:37.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:37.530 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2021-05-12T12:54:38.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:38.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2021-05-12T12:54:38.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:38.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2021-05-12T12:54:39.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:39.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2021-05-12T12:54:39.369 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:39.370 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2021-05-12T12:54:39.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:39.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2021-05-12T12:54:40.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:40.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2021-05-12T12:54:40.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:40.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2021-05-12T12:54:41.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:41.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2021-05-12T12:54:41.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:41.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2021-05-12T12:54:42.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:42.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2021-05-12T12:54:43.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:43.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2021-05-12T12:54:43.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:43.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2021-05-12T12:54:44.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:44.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2021-05-12T12:54:44.885 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:44.886 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2021-05-12T12:54:45.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2021-05-12T12:54:45.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.698 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2021-05-12T12:54:45.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:45.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2021-05-12T12:54:46.333 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:46.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2021-05-12T12:54:48.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:48.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2021-05-12T12:54:48.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:48.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2021-05-12T12:54:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2021-05-12T12:54:50.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:50.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2021-05-12T12:54:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:50.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2021-05-12T12:54:50.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:50.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2021-05-12T12:54:51.234 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2021-05-12T12:54:51.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.535 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2021-05-12T12:54:51.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:51.853 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2021-05-12T12:54:52.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:52.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2021-05-12T12:54:52.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:52.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2021-05-12T12:54:53.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:53.817 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2021-05-12T12:54:54.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.118 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2021-05-12T12:54:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2021-05-12T12:54:54.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:54.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2021-05-12T12:54:55.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:55.038 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2021-05-12T12:54:55.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:55.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2021-05-12T12:54:55.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:55.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2021-05-12T12:54:56.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:56.193 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2021-05-12T12:54:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:56.536 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2021-05-12T12:54:57.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:57.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2021-05-12T12:54:57.972 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:57.973 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2021-05-12T12:54:58.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:58.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2021-05-12T12:54:59.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:59.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2021-05-12T12:54:59.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:54:59.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2021-05-12T12:55:00.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:00.522 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2021-05-12T12:55:00.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:00.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2021-05-12T12:55:01.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:01.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2021-05-12T12:55:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:01.500 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2021-05-12T12:55:01.911 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:01.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2021-05-12T12:55:02.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:02.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2021-05-12T12:55:02.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:02.580 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2021-05-12T12:55:03.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:03.724 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2021-05-12T12:55:04.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:04.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2021-05-12T12:55:04.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:04.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2021-05-12T12:55:04.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:04.727 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2021-05-12T12:55:05.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:05.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2021-05-12T12:55:05.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:05.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2021-05-12T12:55:05.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:05.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2021-05-12T12:55:06.825 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:06.826 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2021-05-12T12:55:07.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:07.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2021-05-12T12:55:07.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:07.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2021-05-12T12:55:07.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:07.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2021-05-12T12:55:08.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:08.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2021-05-12T12:55:08.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:08.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2021-05-12T12:55:08.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:08.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2021-05-12T12:55:09.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:09.927 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2021-05-12T12:55:11.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:11.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2021-05-12T12:55:11.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:11.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2021-05-12T12:55:12.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:12.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2021-05-12T12:55:12.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:12.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2021-05-12T12:55:13.202 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:13.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2021-05-12T12:55:13.569 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:13.570 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2021-05-12T12:55:13.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:13.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2021-05-12T12:55:14.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:14.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2021-05-12T12:55:14.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:14.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2021-05-12T12:55:14.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:14.827 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2021-05-12T12:55:15.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:15.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2021-05-12T12:55:16.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:16.431 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2021-05-12T12:55:16.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:16.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2021-05-12T12:55:17.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:17.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2021-05-12T12:55:17.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:17.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2021-05-12T12:55:17.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:17.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2021-05-12T12:55:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:18.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2021-05-12T12:55:18.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:18.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2021-05-12T12:55:18.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:18.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2021-05-12T12:55:20.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:20.083 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2021-05-12T12:55:20.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:20.405 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2021-05-12T12:55:20.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:20.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2021-05-12T12:55:21.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:21.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2021-05-12T12:55:22.036 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:22.037 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2021-05-12T12:55:23.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:23.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2021-05-12T12:55:23.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:23.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2021-05-12T12:55:24.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:24.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2021-05-12T12:55:24.831 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:24.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2021-05-12T12:55:25.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:25.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2021-05-12T12:55:26.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:26.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2021-05-12T12:55:26.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:26.379 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2021-05-12T12:55:26.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:26.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2021-05-12T12:55:27.047 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:27.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2021-05-12T12:55:27.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:27.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2021-05-12T12:55:28.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:28.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2021-05-12T12:55:29.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:29.147 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2021-05-12T12:55:29.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:29.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2021-05-12T12:55:29.840 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:29.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2021-05-12T12:55:30.141 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:30.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2021-05-12T12:55:30.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:30.435 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2021-05-12T12:55:30.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:30.769 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2021-05-12T12:55:31.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:31.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2021-05-12T12:55:32.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:32.307 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2021-05-12T12:55:32.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:32.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2021-05-12T12:55:32.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:32.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2021-05-12T12:55:33.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:33.311 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2021-05-12T12:55:33.695 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:33.696 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2021-05-12T12:55:35.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 2 + 1 2021-05-12T12:55:35.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=3 2021-05-12T12:55:35.384 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-05-12T12:55:35.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-05-12T12:55:35.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:35.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2021-05-12T12:55:35.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:35.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2021-05-12T12:55:36.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:36.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2021-05-12T12:55:37.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:37.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2021-05-12T12:55:38.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:38.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2021-05-12T12:55:38.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:38.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2021-05-12T12:55:38.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:38.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2021-05-12T12:55:39.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:39.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2021-05-12T12:55:40.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:40.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2021-05-12T12:55:41.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:41.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2021-05-12T12:55:41.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:41.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2021-05-12T12:55:42.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:42.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2021-05-12T12:55:42.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:42.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2021-05-12T12:55:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:42.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2021-05-12T12:55:43.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:43.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2021-05-12T12:55:44.148 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:44.149 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2021-05-12T12:55:44.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:44.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2021-05-12T12:55:44.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:44.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2021-05-12T12:55:45.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:45.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2021-05-12T12:55:47.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:47.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2021-05-12T12:55:47.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:47.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2021-05-12T12:55:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:48.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2021-05-12T12:55:48.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:48.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2021-05-12T12:55:48.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:48.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2021-05-12T12:55:49.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:49.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2021-05-12T12:55:50.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:50.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2021-05-12T12:55:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:50.652 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2021-05-12T12:55:51.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:51.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2021-05-12T12:55:51.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:51.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2021-05-12T12:55:51.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:51.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2021-05-12T12:55:52.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:52.226 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2021-05-12T12:55:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:53.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2021-05-12T12:55:53.469 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:53.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2021-05-12T12:55:53.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:53.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2021-05-12T12:55:54.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:54.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2021-05-12T12:55:54.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:54.433 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2021-05-12T12:55:54.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:54.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2021-05-12T12:55:55.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:55.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2021-05-12T12:55:56.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:56.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2021-05-12T12:55:56.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:56.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2021-05-12T12:55:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2021-05-12T12:55:57.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:57.426 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2021-05-12T12:55:57.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:57.802 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2021-05-12T12:55:59.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:59.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2021-05-12T12:55:59.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:55:59.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2021-05-12T12:56:00.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:00.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2021-05-12T12:56:00.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:00.393 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2021-05-12T12:56:00.702 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:00.703 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2021-05-12T12:56:01.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:01.054 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2021-05-12T12:56:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:01.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2021-05-12T12:56:02.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:02.532 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2021-05-12T12:56:02.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:02.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2021-05-12T12:56:03.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:03.227 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2021-05-12T12:56:03.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:03.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2021-05-12T12:56:03.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:03.855 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2021-05-12T12:56:04.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:04.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2021-05-12T12:56:04.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:04.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2021-05-12T12:56:05.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:05.684 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2021-05-12T12:56:05.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:05.986 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2021-05-12T12:56:06.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:06.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2021-05-12T12:56:07.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:07.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2021-05-12T12:56:07.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:07.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2021-05-12T12:56:08.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:08.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2021-05-12T12:56:08.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:08.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2021-05-12T12:56:09.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:09.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2021-05-12T12:56:10.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:10.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2021-05-12T12:56:11.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:11.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2021-05-12T12:56:11.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:11.865 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2021-05-12T12:56:12.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:12.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2021-05-12T12:56:12.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:12.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2021-05-12T12:56:12.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:12.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2021-05-12T12:56:13.138 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:13.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2021-05-12T12:56:13.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:13.495 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2021-05-12T12:56:14.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:14.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2021-05-12T12:56:14.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:14.874 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2021-05-12T12:56:15.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:15.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2021-05-12T12:56:15.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:15.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2021-05-12T12:56:15.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:15.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2021-05-12T12:56:16.279 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:16.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2021-05-12T12:56:16.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:16.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2021-05-12T12:56:17.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:17.667 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.320 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2021-05-12T12:56:18.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:18.687 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2021-05-12T12:56:19.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:19.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2021-05-12T12:56:19.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:19.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2021-05-12T12:56:19.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:19.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2021-05-12T12:56:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:20.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2021-05-12T12:56:22.127 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:22.128 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2021-05-12T12:56:22.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:22.674 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2021-05-12T12:56:22.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:22.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2021-05-12T12:56:23.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:23.753 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2021-05-12T12:56:24.070 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:24.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2021-05-12T12:56:24.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:24.397 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2021-05-12T12:56:24.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:24.723 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2021-05-12T12:56:25.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:25.041 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2021-05-12T12:56:25.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:25.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2021-05-12T12:56:26.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:26.020 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2021-05-12T12:56:26.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:26.988 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2021-05-12T12:56:27.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:27.339 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2021-05-12T12:56:27.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:27.741 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2021-05-12T12:56:28.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:28.042 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2021-05-12T12:56:28.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:28.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2021-05-12T12:56:28.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:28.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2021-05-12T12:56:28.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:28.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2021-05-12T12:56:30.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:30.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2021-05-12T12:56:30.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:30.726 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2021-05-12T12:56:31.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:31.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2021-05-12T12:56:32.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:32.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2021-05-12T12:56:32.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:32.768 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2021-05-12T12:56:33.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:33.085 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2021-05-12T12:56:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:33.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2021-05-12T12:56:34.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:34.198 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2021-05-12T12:56:34.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:34.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2021-05-12T12:56:34.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:34.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2021-05-12T12:56:35.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:35.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2021-05-12T12:56:36.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:36.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2021-05-12T12:56:36.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:36.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2021-05-12T12:56:36.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:36.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2021-05-12T12:56:37.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:37.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2021-05-12T12:56:37.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:37.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2021-05-12T12:56:37.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:37.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2021-05-12T12:56:38.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:38.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2021-05-12T12:56:39.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:39.322 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2021-05-12T12:56:39.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:39.656 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2021-05-12T12:56:39.983 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:39.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2021-05-12T12:56:40.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:40.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2021-05-12T12:56:40.594 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:40.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2021-05-12T12:56:40.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:40.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2021-05-12T12:56:41.313 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:41.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2021-05-12T12:56:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:42.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2021-05-12T12:56:42.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:42.676 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2021-05-12T12:56:43.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:43.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2021-05-12T12:56:43.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:43.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2021-05-12T12:56:43.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:43.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2021-05-12T12:56:45.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:45.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2021-05-12T12:56:46.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:46.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2021-05-12T12:56:46.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:46.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2021-05-12T12:56:46.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:46.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2021-05-12T12:56:47.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:47.340 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2021-05-12T12:56:48.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:48.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2021-05-12T12:56:48.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:48.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2021-05-12T12:56:48.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:48.887 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2021-05-12T12:56:49.346 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:49.347 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2021-05-12T12:56:49.714 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:49.715 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2021-05-12T12:56:50.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:50.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2021-05-12T12:56:51.300 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:51.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2021-05-12T12:56:51.685 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:51.686 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2021-05-12T12:56:52.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:52.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2021-05-12T12:56:52.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:52.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2021-05-12T12:56:52.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:52.890 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2021-05-12T12:56:53.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:53.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2021-05-12T12:56:54.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:54.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2021-05-12T12:56:54.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2021-05-12T12:56:55.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:55.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2021-05-12T12:56:55.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:55.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2021-05-12T12:56:55.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:55.949 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2021-05-12T12:56:57.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:57.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2021-05-12T12:56:58.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:58.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2021-05-12T12:56:59.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:56:59.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2021-05-12T12:57:00.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:00.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2021-05-12T12:57:00.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:00.318 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2021-05-12T12:57:00.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:00.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2021-05-12T12:57:01.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:01.087 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2021-05-12T12:57:01.546 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:01.547 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2021-05-12T12:57:01.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:01.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2021-05-12T12:57:02.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:02.325 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2021-05-12T12:57:03.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:03.427 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2021-05-12T12:57:03.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:03.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2021-05-12T12:57:04.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:04.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2021-05-12T12:57:04.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:04.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2021-05-12T12:57:04.808 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:04.809 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2021-05-12T12:57:05.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:05.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2021-05-12T12:57:05.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:05.494 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2021-05-12T12:57:06.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:06.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2021-05-12T12:57:06.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:06.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2021-05-12T12:57:07.264 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:07.265 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2021-05-12T12:57:07.709 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:07.710 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2021-05-12T12:57:08.018 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:08.019 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2021-05-12T12:57:08.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:08.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2021-05-12T12:57:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:08.704 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2021-05-12T12:57:09.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:09.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2021-05-12T12:57:11.123 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:11.124 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2021-05-12T12:57:11.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:11.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2021-05-12T12:57:12.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:12.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2021-05-12T12:57:12.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:12.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2021-05-12T12:57:13.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:13.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2021-05-12T12:57:13.475 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:13.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2021-05-12T12:57:13.871 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:13.872 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2021-05-12T12:57:14.179 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:14.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2021-05-12T12:57:14.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:14.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2021-05-12T12:57:14.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:14.832 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2021-05-12T12:57:15.157 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:15.158 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2021-05-12T12:57:16.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-05-12T12:57:16.285 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2021-05-12T12:57:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 3 + 1 2021-05-12T12:57:16.662 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=4 2021-05-12T12:57:16.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: get_not_primary test1 obj1-p1 2021-05-12T12:57:16.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local poolname=test1 2021-05-12T12:57:16.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local objectname=obj1-p1 2021-05-12T12:57:16.664 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: get_primary test1 obj1-p1 2021-05-12T12:57:16.664 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local poolname=test1 2021-05-12T12:57:16.665 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: local objectname=obj1-p1 2021-05-12T12:57:16.665 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:57:16.665 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: jq .acting_primary 2021-05-12T12:57:17.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local primary=1 2021-05-12T12:57:17.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T12:57:17.039 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1173: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-05-12T12:57:17.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: local otherosd=0 2021-05-12T12:57:17.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:145: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:57:17.741 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:57:17.741 INFO:tasks.workunit.client.0.smithi090.stdout:7.0 198 0 0 0 0 2069889024 0 0 592 592 active+clean 2021-05-12T12:50:46.242006+0000 91'592 91:641 [1] 1 [1] 1 0'0 2021-05-12T12:50:16.255616+0000 0'0 2021-05-12T12:50:16.255616+0000 0 2021-05-12T12:57:17.741 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:50:04.115463+0000 91'600 91:666 [1,0] 1 [1,0] 1 0'0 2021-05-12T12:50:01.133820+0000 0'0 2021-05-12T12:50:01.133820+0000 0 2021-05-12T12:57:17.742 INFO:tasks.workunit.client.0.smithi090.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:50:45.199327+0000 91'600 91:666 [1] 1 [1] 1 0'0 2021-05-12T12:50:06.202217+0000 0'0 2021-05-12T12:50:06.202217+0000 0 2021-05-12T12:57:17.742 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:57:17.742 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:57:17.743 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:57:17.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: ERRORS=0 2021-05-12T12:57:17.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph osd set norecover 2021-05-12T12:57:18.467 INFO:tasks.workunit.client.0.smithi090.stderr:norecover is set 2021-05-12T12:57:18.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ceph osd set noout 2021-05-12T12:57:19.238 INFO:tasks.workunit.client.0.smithi090.stderr:noout is set 2021-05-12T12:57:19.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:153: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-05-12T12:57:19.967 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 7 size to 2 2021-05-12T12:57:19.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:154: TEST_recovery_priority: sleep 2 2021-05-12T12:57:21.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:21.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:21.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:21.980 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:21.980 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:21.980 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:21.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:21.981 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:22.080 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:22.080 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:22.081 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:22.081 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:22.081 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:22.081 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:22.081 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:22.082 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:22.082 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:22.082 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:22.082 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:22.083 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:22.083 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:22.083 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:22.083 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:22.083 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:22.084 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:22.084 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:22.084 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:22.084 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:22.090 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: seq 1 10 2021-05-12T12:57:22.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:57:22.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:57:22.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:57:22.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 1 = 10 ']' 2021-05-12T12:57:22.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-05-12T12:57:24.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:57:24.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:57:24.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:57:24.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 2 = 10 ']' 2021-05-12T12:57:24.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-05-12T12:57:26.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:57:26.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:57:26.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:57:27.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: break 2021-05-12T12:57:27.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:169: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:57:27.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:27.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:27.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:27.525 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:57:27.525 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:57:27.525 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:57:27.525 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:57:27.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:27.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:27.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:27.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836584 2021-05-12T12:57:27.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836584 2021-05-12T12:57:27.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584' 2021-05-12T12:57:27.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:27.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:27.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705766 2021-05-12T12:57:27.999 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705766 2021-05-12T12:57:28.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705766' 2021-05-12T12:57:28.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:28.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542245 2021-05-12T12:57:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542245 2021-05-12T12:57:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705766 2-60129542245' 2021-05-12T12:57:28.125 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:28.126 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:28.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411426 2021-05-12T12:57:28.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411426 2021-05-12T12:57:28.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705766 2-60129542245 3-77309411426' 2021-05-12T12:57:28.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:28.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:28.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247904 2021-05-12T12:57:28.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247904 2021-05-12T12:57:28.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836584 1-38654705766 2-60129542245 3-77309411426 4-98784247904' 2021-05-12T12:57:28.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:28.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836584 2021-05-12T12:57:28.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:28.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:28.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836584 2021-05-12T12:57:28.383 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:28.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836584 2021-05-12T12:57:28.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2021-05-12T12:57:28.383 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836584 2021-05-12T12:57:28.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:28.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836582 -lt 21474836584 2021-05-12T12:57:28.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:29.737 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:29.738 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:30.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836584 -lt 21474836584 2021-05-12T12:57:30.106 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:30.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705766 2021-05-12T12:57:30.107 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:30.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:30.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705766 2021-05-12T12:57:30.108 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:30.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705766 2021-05-12T12:57:30.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705766' 2021-05-12T12:57:30.110 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705766 2021-05-12T12:57:30.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:30.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705766 -lt 38654705766 2021-05-12T12:57:30.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:30.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542245 2021-05-12T12:57:30.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:30.489 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:30.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542245 2021-05-12T12:57:30.489 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:30.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542245 2021-05-12T12:57:30.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542245' 2021-05-12T12:57:30.491 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542245 2021-05-12T12:57:30.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:30.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542245 -lt 60129542245 2021-05-12T12:57:30.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:30.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411426 2021-05-12T12:57:30.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:30.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:30.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411426 2021-05-12T12:57:30.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:30.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411426 2021-05-12T12:57:30.852 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411426' 2021-05-12T12:57:30.853 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411426 2021-05-12T12:57:30.853 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:31.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411426 -lt 77309411426 2021-05-12T12:57:31.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:31.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247904 2021-05-12T12:57:31.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:31.246 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:31.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247904 2021-05-12T12:57:31.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:31.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247904 2021-05-12T12:57:31.247 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247904' 2021-05-12T12:57:31.248 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247904 2021-05-12T12:57:31.248 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:31.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247904 -lt 98784247904 2021-05-12T12:57:31.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:31.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:31.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:31.607 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:31.608 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:31.608 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:31.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:31.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:31.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:31.698 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:31.699 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:31.699 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:31.699 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:31.700 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:31.700 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:31.700 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:31.700 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:31.700 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:31.701 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:31.701 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:31.701 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:31.701 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:31.701 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:31.702 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:31.702 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:31.702 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:31.702 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:31.702 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:31.703 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:31.706 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: ceph osd out osd.0 2021-05-12T12:57:32.205 INFO:tasks.workunit.client.0.smithi090.stderr:osd.0 is already out. 2021-05-12T12:57:32.214 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: sleep 2 2021-05-12T12:57:34.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:174: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:57:34.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:34.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:34.585 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:34.585 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:57:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:57:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:57:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:57:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:34.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:34.587 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836587 2021-05-12T12:57:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836587 2021-05-12T12:57:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836587' 2021-05-12T12:57:34.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:34.712 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705769 2021-05-12T12:57:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705769 2021-05-12T12:57:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836587 1-38654705769' 2021-05-12T12:57:34.839 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:34.839 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:34.964 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542248 2021-05-12T12:57:34.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542248 2021-05-12T12:57:34.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836587 1-38654705769 2-60129542248' 2021-05-12T12:57:34.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:34.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:35.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411429 2021-05-12T12:57:35.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411429 2021-05-12T12:57:35.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836587 1-38654705769 2-60129542248 3-77309411429' 2021-05-12T12:57:35.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:35.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:35.251 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247907 2021-05-12T12:57:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247907 2021-05-12T12:57:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836587 1-38654705769 2-60129542248 3-77309411429 4-98784247907' 2021-05-12T12:57:35.252 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:35.252 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836587 2021-05-12T12:57:35.253 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:35.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:35.254 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836587 2021-05-12T12:57:35.254 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:35.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836587 2021-05-12T12:57:35.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2021-05-12T12:57:35.255 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836587 2021-05-12T12:57:35.255 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:35.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836587 -lt 21474836587 2021-05-12T12:57:35.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:35.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705769 2021-05-12T12:57:35.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:35.617 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:35.617 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705769 2021-05-12T12:57:35.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:35.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705769 2021-05-12T12:57:35.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705769' 2021-05-12T12:57:35.619 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705769 2021-05-12T12:57:35.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:36.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705770 -lt 38654705769 2021-05-12T12:57:36.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:36.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542248 2021-05-12T12:57:36.005 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:36.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:36.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542248 2021-05-12T12:57:36.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:36.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542248 2021-05-12T12:57:36.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542248' 2021-05-12T12:57:36.007 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542248 2021-05-12T12:57:36.008 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:36.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542248 -lt 60129542248 2021-05-12T12:57:36.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:36.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411429 2021-05-12T12:57:36.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:36.385 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:36.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411429 2021-05-12T12:57:36.385 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:36.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411429 2021-05-12T12:57:36.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411429' 2021-05-12T12:57:36.387 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411429 2021-05-12T12:57:36.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:36.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411429 -lt 77309411429 2021-05-12T12:57:36.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:36.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247907 2021-05-12T12:57:36.743 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:36.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:36.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247907 2021-05-12T12:57:36.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:36.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247907 2021-05-12T12:57:36.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247907' 2021-05-12T12:57:36.745 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247907 2021-05-12T12:57:36.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:37.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247905 -lt 98784247907 2021-05-12T12:57:37.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:38.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:38.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:38.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247908 -lt 98784247907 2021-05-12T12:57:38.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:38.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:38.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:38.472 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:38.473 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:38.473 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:38.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:38.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:38.474 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:38.564 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:38.564 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:38.564 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:38.564 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:38.565 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:38.565 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:38.565 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:38.565 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:57:38.565 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:38.566 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:38.566 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:38.566 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:38.566 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:38.566 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:38.567 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:38.567 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:38.567 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:38.567 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:38.568 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:38.568 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:38.568 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:38.568 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:38.568 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:38.569 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:38.569 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:38.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:57:38.904 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:57:38.904 INFO:tasks.workunit.client.0.smithi090.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-05-12T12:57:27.056947+0000 91'600 103:670 [1,4] 1 [1] 1 0'0 2021-05-12T12:50:16.255616+0000 0'0 2021-05-12T12:50:16.255616+0000 0 2021-05-12T12:57:38.905 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 0 200 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:57:33.793838+0000 91'600 104:686 [1,2] 1 [1] 1 0'0 2021-05-12T12:50:01.133820+0000 0'0 2021-05-12T12:50:01.133820+0000 0 2021-05-12T12:57:38.905 INFO:tasks.workunit.client.0.smithi090.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T12:50:45.199327+0000 91'600 103:678 [1] 1 [1] 1 0'0 2021-05-12T12:50:06.202217+0000 0'0 2021-05-12T12:50:06.202217+0000 0 2021-05-12T12:57:38.905 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:57:38.905 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:57:38.906 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:57:38.915 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-05-12T12:57:39.599 INFO:tasks.workunit.client.0.smithi090.stderr:set pool 3 size to 2 2021-05-12T12:57:39.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: sleep 2 2021-05-12T12:57:41.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:180: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:57:41.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:41.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:57:41.977 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:57:41.978 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:57:41.978 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:57:41.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:41.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:41.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:42.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836591 2021-05-12T12:57:42.136 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836591 2021-05-12T12:57:42.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836591' 2021-05-12T12:57:42.137 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:42.137 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:42.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705773 2021-05-12T12:57:42.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705773 2021-05-12T12:57:42.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836591 1-38654705773' 2021-05-12T12:57:42.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:42.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542252 2021-05-12T12:57:42.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542252 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836591 1-38654705773 2-60129542252' 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:42.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:42.514 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411433 2021-05-12T12:57:42.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411433 2021-05-12T12:57:42.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836591 1-38654705773 2-60129542252 3-77309411433' 2021-05-12T12:57:42.515 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:42.515 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:42.642 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247911 2021-05-12T12:57:42.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247911 2021-05-12T12:57:42.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836591 1-38654705773 2-60129542252 3-77309411433 4-98784247911' 2021-05-12T12:57:42.643 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:42.643 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836591 2021-05-12T12:57:42.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:42.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:42.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836591 2021-05-12T12:57:42.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:42.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836591 2021-05-12T12:57:42.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2021-05-12T12:57:42.646 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836591 2021-05-12T12:57:42.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:43.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836589 -lt 21474836591 2021-05-12T12:57:43.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:44.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:44.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:44.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836591 -lt 21474836591 2021-05-12T12:57:44.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:44.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705773 2021-05-12T12:57:44.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:44.382 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:44.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705773 2021-05-12T12:57:44.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:44.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705773 2021-05-12T12:57:44.383 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705773' 2021-05-12T12:57:44.384 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705773 2021-05-12T12:57:44.384 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:44.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705773 -lt 38654705773 2021-05-12T12:57:44.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:44.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542252 2021-05-12T12:57:44.765 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:44.766 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:44.766 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542252 2021-05-12T12:57:44.766 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:44.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542252 2021-05-12T12:57:44.767 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542252' 2021-05-12T12:57:44.768 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542252 2021-05-12T12:57:44.768 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:45.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542252 -lt 60129542252 2021-05-12T12:57:45.182 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:45.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411433 2021-05-12T12:57:45.183 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:45.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:45.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411433 2021-05-12T12:57:45.184 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:45.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411433 2021-05-12T12:57:45.185 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411433' 2021-05-12T12:57:45.185 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411433 2021-05-12T12:57:45.186 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:45.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411433 -lt 77309411433 2021-05-12T12:57:45.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:45.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247911 2021-05-12T12:57:45.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:45.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:45.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247911 2021-05-12T12:57:45.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:45.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247911 2021-05-12T12:57:45.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247911' 2021-05-12T12:57:45.568 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247911 2021-05-12T12:57:45.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:45.955 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247911 -lt 98784247911 2021-05-12T12:57:45.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:45.956 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:45.957 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:45.957 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:45.958 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:45.958 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:45.959 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:45.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:45.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:46.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:46.058 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:46.059 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:46.059 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:46.059 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:46.059 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:46.059 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:46.060 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:46.060 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:57:46.060 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:46.060 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:46.061 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:46.061 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:46.061 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:57:46.061 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:46.061 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:46.062 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:46.062 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:46.062 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:46.062 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:46.063 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:46.063 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:46.063 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:46.063 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:46.063 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:46.064 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:46.064 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:46.064 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:46.064 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:46.064 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:46.065 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:46.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: ceph pg dump pgs 2021-05-12T12:57:46.452 INFO:tasks.workunit.client.0.smithi090.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T12:57:46.453 INFO:tasks.workunit.client.0.smithi090.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-05-12T12:57:27.056947+0000 91'600 108:675 [1,4] 1 [1] 1 0'0 2021-05-12T12:50:16.255616+0000 0'0 2021-05-12T12:50:16.255616+0000 0 2021-05-12T12:57:46.453 INFO:tasks.workunit.client.0.smithi090.stdout:1.0 200 0 0 200 0 2097152000 0 0 600 600 active+recovering+undersized+remapped 2021-05-12T12:57:40.555642+0000 91'600 108:691 [1,2] 1 [1] 1 0'0 2021-05-12T12:50:01.133820+0000 0'0 2021-05-12T12:50:01.133820+0000 0 2021-05-12T12:57:46.453 INFO:tasks.workunit.client.0.smithi090.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T12:57:41.763832+0000 91'600 109:691 [1,2] 1 [1] 1 0'0 2021-05-12T12:50:06.202217+0000 0'0 2021-05-12T12:50:06.202217+0000 0 2021-05-12T12:57:46.453 INFO:tasks.workunit.client.0.smithi090.stdout: 2021-05-12T12:57:46.454 INFO:tasks.workunit.client.0.smithi090.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T12:57:46.454 INFO:tasks.workunit.client.0.smithi090.stderr:dumped pgs 2021-05-12T12:57:46.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:46.464 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-05-12T12:57:46.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: PRIO= 2021-05-12T12:57:46.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:57:46.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: echo 'The normal PG 1.0 doesn'\''t have prio 190 queued waiting' 2021-05-12T12:57:46.467 INFO:tasks.workunit.client.0.smithi090.stdout:The normal PG 1.0 doesn't have prio 190 queued waiting 2021-05-12T12:57:46.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: expr 0 + 1 2021-05-12T12:57:46.468 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: ERRORS=1 2021-05-12T12:57:46.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:46.469 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:57:46.470 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:57:46.471 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:57:46.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:194: TEST_recovery_priority: '[' 1.0 '!=' 7.0 ']' 2021-05-12T12:57:46.471 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: echo 'The first force-recovery PG 7.0 didn'\''t become the in progress item' 2021-05-12T12:57:46.471 INFO:tasks.workunit.client.0.smithi090.stdout:The first force-recovery PG 7.0 didn't become the in progress item 2021-05-12T12:57:46.472 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: expr 1 + 1 2021-05-12T12:57:46.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: ERRORS=2 2021-05-12T12:57:46.473 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: seq 1 10 2021-05-12T12:57:46.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-05-12T12:57:46.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: ceph pg force-recovery 3.0 2021-05-12T12:57:46.473 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-05-12T12:57:46.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: break 2021-05-12T12:57:46.823 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:219: TEST_recovery_priority: sleep 2 2021-05-12T12:57:48.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:48.911 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:48.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:48.912 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:48.912 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:48.913 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:48.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:48.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:48.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:48.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:221: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:48.936 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:48.936 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:48.937 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:48.937 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:48.937 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:48.937 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:48.938 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:48.938 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:57:48.938 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:48.938 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:48.938 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:48.939 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:48.939 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:57:48.939 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:48.939 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:48.940 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:48.940 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:48.940 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:48.940 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:48.941 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:48.941 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:48.941 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:48.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:48.941 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:48.942 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:48.942 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:48.942 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:48.942 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:48.943 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:48.943 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:48.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:48.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-05-12T12:57:48.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: PRIO= 2021-05-12T12:57:48.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: '[' '' '!=' 255 ']' 2021-05-12T12:57:48.945 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: echo 'The second force-recovery PG 3.0 doesn'\''t have prio 255' 2021-05-12T12:57:48.945 INFO:tasks.workunit.client.0.smithi090.stdout:The second force-recovery PG 3.0 doesn't have prio 255 2021-05-12T12:57:48.946 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: expr 2 + 1 2021-05-12T12:57:48.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: ERRORS=3 2021-05-12T12:57:48.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:228: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:57:48.947 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:57:48.947 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:57:49.326 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:57:49.326 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:57:49.326 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:57:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:57:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:57:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:57:49.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:49.328 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:57:49.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836594 2021-05-12T12:57:49.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836594 2021-05-12T12:57:49.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836594' 2021-05-12T12:57:49.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:49.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:57:49.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705776 2021-05-12T12:57:49.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705776 2021-05-12T12:57:49.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836594 1-38654705776' 2021-05-12T12:57:49.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:49.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:57:49.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542255 2021-05-12T12:57:49.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542255 2021-05-12T12:57:49.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836594 1-38654705776 2-60129542255' 2021-05-12T12:57:49.738 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:49.739 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:57:49.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411436 2021-05-12T12:57:49.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411436 2021-05-12T12:57:49.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836594 1-38654705776 2-60129542255 3-77309411436' 2021-05-12T12:57:49.864 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:57:49.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:57:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247914 2021-05-12T12:57:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247914 2021-05-12T12:57:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836594 1-38654705776 2-60129542255 3-77309411436 4-98784247914' 2021-05-12T12:57:49.989 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:49.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836594 2021-05-12T12:57:49.990 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:49.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:57:49.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836594 2021-05-12T12:57:49.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:49.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836594 2021-05-12T12:57:49.992 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2021-05-12T12:57:49.992 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836594 2021-05-12T12:57:49.993 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:50.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836592 -lt 21474836594 2021-05-12T12:57:50.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:57:51.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:57:51.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:57:51.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836595 -lt 21474836594 2021-05-12T12:57:51.745 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:51.745 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705776 2021-05-12T12:57:51.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:51.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:57:51.746 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705776 2021-05-12T12:57:51.747 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:51.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705776 2021-05-12T12:57:51.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705776' 2021-05-12T12:57:51.748 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705776 2021-05-12T12:57:51.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:57:52.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705777 -lt 38654705776 2021-05-12T12:57:52.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:52.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542255 2021-05-12T12:57:52.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:52.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:57:52.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542255 2021-05-12T12:57:52.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:52.104 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542255 2021-05-12T12:57:52.105 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542255' 2021-05-12T12:57:52.105 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542255 2021-05-12T12:57:52.105 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:57:52.464 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542256 -lt 60129542255 2021-05-12T12:57:52.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:52.465 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411436 2021-05-12T12:57:52.465 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:52.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:57:52.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411436 2021-05-12T12:57:52.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:52.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411436 2021-05-12T12:57:52.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411436' 2021-05-12T12:57:52.468 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411436 2021-05-12T12:57:52.468 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:57:52.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411437 -lt 77309411436 2021-05-12T12:57:52.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:57:52.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247914 2021-05-12T12:57:52.881 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:57:52.882 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:57:52.882 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247914 2021-05-12T12:57:52.882 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:57:52.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247914 2021-05-12T12:57:52.883 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247914' 2021-05-12T12:57:52.884 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247914 2021-05-12T12:57:52.884 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:57:53.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247915 -lt 98784247914 2021-05-12T12:57:53.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:232: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2021-05-12T12:57:53.627 INFO:tasks.workunit.client.0.smithi090.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2021-05-12T12:57:53.627 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T12:57:53.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:233: TEST_recovery_priority: sleep 2 2021-05-12T12:57:55.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:57:55.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:57:55.639 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:57:55.640 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:57:55.640 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:57:55.640 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:57:55.640 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:57:55.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:57:55.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:57:55.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:55.751 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:57:55.751 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:57:55.751 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:55.751 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:55.752 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:55.752 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:57:55.752 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:55.752 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:57:55.753 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:55.753 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:55.753 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:55.753 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:55.754 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:57:55.754 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:57:55.754 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:55.754 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:55.754 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:57:55.755 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:57:55.755 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:57:55.755 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:57:55.755 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:55.756 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:57:55.756 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:57:55.756 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:57:55.756 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:57:55.756 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:57:55.757 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:57:55.757 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:57:55.757 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:57:55.757 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:57:55.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:55.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-05-12T12:57:55.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: PRIO= 2021-05-12T12:57:55.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:57:55.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: echo 'After cancel-recovery PG 7.0 doesn'\''t have prio 190' 2021-05-12T12:57:55.760 INFO:tasks.workunit.client.0.smithi090.stdout:After cancel-recovery PG 7.0 doesn't have prio 190 2021-05-12T12:57:55.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: expr 3 + 1 2021-05-12T12:57:55.761 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: ERRORS=4 2021-05-12T12:57:55.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:57:55.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:57:55.762 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:57:55.762 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:57:55.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:245: TEST_recovery_priority: '[' 1.0 '!=' 3.0 ']' 2021-05-12T12:57:55.763 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: echo 'The force-recovery PG 3.0 didn'\''t become the in progress item' 2021-05-12T12:57:55.763 INFO:tasks.workunit.client.0.smithi090.stdout:The force-recovery PG 3.0 didn't become the in progress item 2021-05-12T12:57:55.764 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: expr 4 + 1 2021-05-12T12:57:55.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: ERRORS=5 2021-05-12T12:57:55.764 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:258: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2021-05-12T12:57:56.100 INFO:tasks.workunit.client.0.smithi090.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2021-05-12T12:57:56.101 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T12:57:56.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:259: TEST_recovery_priority: sleep 5 2021-05-12T12:58:01.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:58:01.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:58:01.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:58:01.113 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:58:01.113 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:58:01.113 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:58:01.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:58:01.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:58:01.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:01.215 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:01.215 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:01.215 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:01.215 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:01.216 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:01.216 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:01.216 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:01.216 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:01.216 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:01.218 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:01.218 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:01.218 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:01.218 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:01.218 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:01.220 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:01.220 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:01.220 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:01.220 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:01.220 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:01.221 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:01.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: flush_pg_stats 2021-05-12T12:58:01.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:58:01.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:58:01.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:58:01.582 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:58:01.582 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:58:01.582 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:58:01.583 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:58:01.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:58:01.583 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:01.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:58:01.733 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836599 2021-05-12T12:58:01.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836599 2021-05-12T12:58:01.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836599' 2021-05-12T12:58:01.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:01.734 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:58:01.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705781 2021-05-12T12:58:01.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705781 2021-05-12T12:58:01.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836599 1-38654705781' 2021-05-12T12:58:01.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:01.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:58:01.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542260 2021-05-12T12:58:01.984 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542260 2021-05-12T12:58:01.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836599 1-38654705781 2-60129542260' 2021-05-12T12:58:01.985 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:01.985 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:58:02.110 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411441 2021-05-12T12:58:02.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411441 2021-05-12T12:58:02.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836599 1-38654705781 2-60129542260 3-77309411441' 2021-05-12T12:58:02.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:02.111 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:58:02.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247919 2021-05-12T12:58:02.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247919 2021-05-12T12:58:02.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836599 1-38654705781 2-60129542260 3-77309411441 4-98784247919' 2021-05-12T12:58:02.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:02.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836599 2021-05-12T12:58:02.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:02.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:58:02.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836599 2021-05-12T12:58:02.239 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:02.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836599 2021-05-12T12:58:02.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2021-05-12T12:58:02.240 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836599 2021-05-12T12:58:02.241 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:58:02.595 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836597 -lt 21474836599 2021-05-12T12:58:02.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:58:03.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:58:03.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:58:03.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836599 -lt 21474836599 2021-05-12T12:58:03.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:03.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705781 2021-05-12T12:58:03.965 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:03.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:58:03.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705781 2021-05-12T12:58:03.967 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:03.967 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705781 2021-05-12T12:58:03.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705781' 2021-05-12T12:58:03.968 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705781 2021-05-12T12:58:03.969 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:58:04.376 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705781 -lt 38654705781 2021-05-12T12:58:04.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:04.377 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542260 2021-05-12T12:58:04.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:04.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:58:04.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542260 2021-05-12T12:58:04.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:04.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542260 2021-05-12T12:58:04.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542260' 2021-05-12T12:58:04.380 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542260 2021-05-12T12:58:04.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:58:04.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542260 -lt 60129542260 2021-05-12T12:58:04.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:04.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411441 2021-05-12T12:58:04.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:04.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:58:04.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411441 2021-05-12T12:58:04.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:04.757 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411441 2021-05-12T12:58:04.757 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411441 2021-05-12T12:58:04.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411441' 2021-05-12T12:58:04.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:58:05.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411441 -lt 77309411441 2021-05-12T12:58:05.116 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:05.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247919 2021-05-12T12:58:05.117 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:05.117 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:58:05.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247919 2021-05-12T12:58:05.118 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:05.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247919 2021-05-12T12:58:05.119 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247919' 2021-05-12T12:58:05.119 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247919 2021-05-12T12:58:05.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:58:05.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247919 -lt 98784247919 2021-05-12T12:58:05.505 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:264: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-05-12T12:58:05.844 INFO:tasks.workunit.client.0.smithi090.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2021-05-12T12:58:05.845 INFO:tasks.workunit.client.0.smithi090.stderr: 2021-05-12T12:58:05.854 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:265: TEST_recovery_priority: sleep 2 2021-05-12T12:58:07.856 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:58:07.857 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:58:07.857 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:58:07.857 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:58:07.857 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:58:07.858 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:58:07.858 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:58:07.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: CEPH_ARGS= 2021-05-12T12:58:07.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:07.968 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:58:07.969 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:07.969 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:07.969 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:07.969 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:07.970 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:07.970 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:07.970 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:07.970 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:07.970 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:07.971 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:07.971 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:07.971 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:07.971 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:07.972 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:07.972 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:07.972 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:07.972 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:07.972 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:07.973 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:07.974 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:07.975 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-05-12T12:58:07.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: PRIO= 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: '[' '' '!=' 190 ']' 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: echo 'After cancel-force-recovery PG 7.0 doesn'\''t have prio 190' 2021-05-12T12:58:07.977 INFO:tasks.workunit.client.0.smithi090.stdout:After cancel-force-recovery PG 7.0 doesn't have prio 190 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: expr 5 + 1 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: ERRORS=6 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-05-12T12:58:07.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T12:58:07.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: eval 'ITEM="1.0"' 2021-05-12T12:58:07.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: ITEM=1.0 2021-05-12T12:58:07.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:277: TEST_recovery_priority: '[' 1.0 '!=' 7.0 ']' 2021-05-12T12:58:07.980 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: echo 'The force-recovery PG 7.0 didn'\''t get promoted to an in progress item' 2021-05-12T12:58:07.981 INFO:tasks.workunit.client.0.smithi090.stdout:The force-recovery PG 7.0 didn't get promoted to an in progress item 2021-05-12T12:58:07.981 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: expr 6 + 1 2021-05-12T12:58:07.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: ERRORS=7 2021-05-12T12:58:07.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:290: TEST_recovery_priority: ceph osd unset noout 2021-05-12T12:58:08.599 INFO:tasks.workunit.client.0.smithi090.stderr:noout is unset 2021-05-12T12:58:08.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:291: TEST_recovery_priority: ceph osd unset norecover 2021-05-12T12:58:09.511 INFO:tasks.workunit.client.0.smithi090.stderr:norecover is unset 2021-05-12T12:58:09.522 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: get_asok_path osd.1 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T12:58:09.523 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185878 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185878/ceph-osd.1.asok 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations' 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations' 2021-05-12T12:58:09.524 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T12:58:09.525 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi090.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T12:58:09.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T12:58:09.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T12:58:09.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T12:58:09.612 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T12:58:09.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T12:58:09.613 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T12:58:09.974 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T12:58:09.975 INFO:tasks.workunit.client.0.smithi090.stderr:1 2021-05-12T12:58:09.975 INFO:tasks.workunit.client.0.smithi090.stderr:2 2021-05-12T12:58:09.975 INFO:tasks.workunit.client.0.smithi090.stderr:3 2021-05-12T12:58:09.976 INFO:tasks.workunit.client.0.smithi090.stderr:4' 2021-05-12T12:58:09.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T12:58:09.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:09.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T12:58:10.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836602 2021-05-12T12:58:10.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836602 2021-05-12T12:58:10.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836602' 2021-05-12T12:58:10.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:10.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T12:58:10.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=38654705785 2021-05-12T12:58:10.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 38654705785 2021-05-12T12:58:10.228 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836602 1-38654705785' 2021-05-12T12:58:10.229 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:10.229 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T12:58:10.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=60129542263 2021-05-12T12:58:10.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 60129542263 2021-05-12T12:58:10.354 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836602 1-38654705785 2-60129542263' 2021-05-12T12:58:10.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:10.355 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T12:58:10.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=77309411444 2021-05-12T12:58:10.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 77309411444 2021-05-12T12:58:10.480 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836602 1-38654705785 2-60129542263 3-77309411444' 2021-05-12T12:58:10.481 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T12:58:10.481 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-05-12T12:58:10.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=98784247922 2021-05-12T12:58:10.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 98784247922 2021-05-12T12:58:10.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836602 1-38654705785 2-60129542263 3-77309411444 4-98784247922' 2021-05-12T12:58:10.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:10.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836602 2021-05-12T12:58:10.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:10.608 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T12:58:10.608 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836602 2021-05-12T12:58:10.609 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:10.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836602 2021-05-12T12:58:10.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2021-05-12T12:58:10.610 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.0 seq 21474836602 2021-05-12T12:58:10.610 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:58:11.030 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836600 -lt 21474836602 2021-05-12T12:58:11.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T12:58:12.031 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T12:58:12.032 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T12:58:12.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836603 -lt 21474836602 2021-05-12T12:58:12.396 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:12.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705785 2021-05-12T12:58:12.397 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:12.398 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T12:58:12.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-38654705785 2021-05-12T12:58:12.398 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:12.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=38654705785 2021-05-12T12:58:12.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 38654705785' 2021-05-12T12:58:12.400 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.1 seq 38654705785 2021-05-12T12:58:12.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T12:58:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 38654705785 -lt 38654705785 2021-05-12T12:58:12.932 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:12.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542263 2021-05-12T12:58:12.933 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:12.933 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T12:58:12.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-60129542263 2021-05-12T12:58:12.934 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:12.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=60129542263 2021-05-12T12:58:12.935 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 60129542263' 2021-05-12T12:58:12.936 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.2 seq 60129542263 2021-05-12T12:58:12.936 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T12:58:13.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 60129542264 -lt 60129542263 2021-05-12T12:58:13.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:13.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-77309411444 2021-05-12T12:58:13.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:13.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T12:58:13.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-77309411444 2021-05-12T12:58:13.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:13.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=77309411444 2021-05-12T12:58:13.390 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 77309411444' 2021-05-12T12:58:13.390 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.3 seq 77309411444 2021-05-12T12:58:13.391 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T12:58:13.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 77309411445 -lt 77309411444 2021-05-12T12:58:13.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T12:58:13.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247922 2021-05-12T12:58:13.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T12:58:13.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=4 2021-05-12T12:58:13.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 4-98784247922 2021-05-12T12:58:13.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T12:58:13.815 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=98784247922 2021-05-12T12:58:13.816 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.4 seq 98784247922' 2021-05-12T12:58:13.816 INFO:tasks.workunit.client.0.smithi090.stdout:waiting osd.4 seq 98784247922 2021-05-12T12:58:13.817 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 4 2021-05-12T12:58:14.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 98784247923 -lt 98784247922 2021-05-12T12:58:14.175 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T12:58:14.175 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:14.175 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:14.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T12:58:14.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:14.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:14.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:14.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:14.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:14.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:14.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:15.074 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:15.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:15.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:15.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:15.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T12:58:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T12:58:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T12:58:15.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:15.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:15.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:15.649 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:15.649 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:15.650 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:15.650 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:15.650 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:15.650 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:15.651 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:15.651 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:15.651 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:15.651 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:15.652 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:15.652 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:15.652 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:15.652 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:15.653 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:15.653 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:15.653 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:15.653 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:15.653 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:15.654 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:15.654 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:15.654 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:15.654 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:15.655 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:15.655 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:15.655 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:15.655 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:15.656 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:15.656 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:15.656 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:15.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:15.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:15.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:15.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:15.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:15.761 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:16.120 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:16.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:16.120 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:16.121 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:16.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:16.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:16.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:16.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:16.597 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:16.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:16.598 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:16.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:16.598 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:17.071 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=3209241 2021-05-12T12:58:17.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 3209241 '!=' null 2021-05-12T12:58:17.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:17.072 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:17.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:17.073 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:17.165 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:17.166 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:17.166 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:17.166 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:17.167 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:17.167 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:17.167 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:17.167 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:17.168 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:17.168 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:17.168 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:17.168 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:17.168 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:17.169 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:17.169 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:17.169 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:17.169 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:17.170 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:17.170 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:17.170 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:17.170 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:17.171 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:17.171 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:17.171 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:17.171 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:17.171 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:17.172 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:17.172 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:17.172 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:17.172 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:17.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:17.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:17.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:17.276 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:17.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:17.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:17.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:17.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:17.277 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:17.626 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:17.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:17.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:17.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:18.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:18.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:18.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:18.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:18.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:18.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:18.132 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:18.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:18.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:18.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2620911 2021-05-12T12:58:18.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2620911 '!=' null 2021-05-12T12:58:18.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:18.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:18.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:18.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:18.712 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:18.712 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:18.713 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:18.713 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:18.713 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:18.713 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:18.714 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:18.714 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:18.714 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:18.714 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:18.715 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:18.715 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:18.715 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:18.715 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:18.715 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:18.716 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:18.716 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:18.716 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:18.716 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:18.717 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:18.717 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:18.717 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:18.717 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:18.717 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:18.718 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:18.718 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:18.718 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:18.718 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:18.719 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:18.719 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:18.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:18.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:18.822 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:18.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:18.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:18.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:18.823 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:18.824 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:18.824 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:19.192 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:19.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:19.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:19.193 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:19.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:19.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:19.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:19.683 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:19.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:19.683 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:20.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=2118485 2021-05-12T12:58:20.160 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 2118485 '!=' null 2021-05-12T12:58:20.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:20.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:20.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:20.161 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:20.254 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:20.254 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:20.255 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:20.255 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:20.255 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:20.255 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:20.256 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:20.256 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:20.256 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:20.256 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:20.257 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:20.257 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:20.257 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:20.257 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:20.258 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:20.258 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:20.258 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:20.258 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:20.258 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:20.259 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:20.259 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:20.259 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:20.259 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:20.260 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:20.260 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:20.260 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:20.260 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:20.261 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:20.261 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:20.261 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:20.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:20.363 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:20.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:20.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:20.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:20.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:20.365 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:20.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:20.366 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:20.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:20.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:20.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:20.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:21.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:21.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:21.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:21.224 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:21.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:21.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:21.225 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:21.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:21.225 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:21.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4510506 2021-05-12T12:58:21.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4510506 '!=' null 2021-05-12T12:58:21.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:21.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:21.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:21.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:21.802 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:21.802 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:21.802 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:21.802 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:21.803 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:21.803 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:21.803 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:21.803 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:21.804 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:21.804 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:21.804 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:21.804 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:21.805 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:21.805 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:21.805 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:21.805 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:21.805 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:21.806 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:21.806 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:21.806 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:21.806 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:21.807 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:21.807 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:21.807 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:21.807 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:21.807 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:21.808 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:21.808 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:21.808 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:21.808 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:21.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:21.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:21.913 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:21.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:21.914 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:21.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:21.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:21.915 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:21.916 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:22.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:22.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:22.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:22.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:22.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:22.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:22.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:22.812 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:22.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:22.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:22.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:22.813 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:22.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:23.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368275 2021-05-12T12:58:23.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368275 '!=' null 2021-05-12T12:58:23.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:23.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:23.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:23.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:23.496 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:23.496 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:23.496 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:23.496 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:23.497 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:23.497 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:23.497 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:23.497 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:23.497 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:23.498 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:23.498 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:23.498 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:23.498 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:23.499 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:23.499 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:23.499 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:23.499 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:23.499 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:23.500 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:23.500 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:23.500 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:23.500 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:23.501 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:23.501 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:23.501 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:23.501 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:23.501 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:23.502 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:23.502 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:23.502 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:23.504 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:23.605 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:23.606 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:23.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:23.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:23.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:23.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:23.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:23.607 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:23.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:23.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:23.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:23.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:24.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:24.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:24.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:24.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:24.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:24.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:24.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:24.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:24.447 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:24.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368275 2021-05-12T12:58:24.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368275 '!=' null 2021-05-12T12:58:24.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:24.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:24.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:24.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:25.036 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:25.037 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:25.037 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:25.037 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:25.037 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:25.037 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:25.038 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:25.038 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:25.038 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:25.038 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:25.039 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:25.039 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:25.039 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:25.039 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:25.040 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:25.040 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:25.040 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:25.040 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:25.040 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:25.041 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:25.041 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:25.041 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:25.041 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:25.041 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:25.042 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:25.042 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:25.042 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:25.042 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:25.043 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:25.043 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:25.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:25.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:25.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:25.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:25.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:25.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:25.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:25.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:25.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:25.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:25.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:25.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:25.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:26.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:26.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:26.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:26.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:26.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:26.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:26.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:26.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:26.028 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:26.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989263 2021-05-12T12:58:26.511 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989263 '!=' null 2021-05-12T12:58:26.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:26.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:26.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:26.512 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:26.605 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:26.605 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:26.606 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:26.606 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:26.606 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:26.606 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:26.607 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:26.607 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:26.607 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:26.607 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:26.607 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:26.608 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:26.608 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:26.608 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:26.608 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:26.609 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:26.609 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:26.609 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:26.609 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:26.609 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:26.610 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:26.610 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:26.610 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:26.610 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:26.611 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:26.611 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:26.611 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:26.611 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:26.612 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:26.612 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:26.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:26.716 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:26.717 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:26.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:26.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:26.717 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:26.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:26.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:26.718 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:27.084 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:27.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:27.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:27.084 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:27.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:27.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:27.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:27.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:27.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:27.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:27.572 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:27.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:27.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:28.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241950 2021-05-12T12:58:28.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241950 '!=' null 2021-05-12T12:58:28.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:28.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:28.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:28.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:28.143 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:28.144 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:28.144 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:28.144 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:28.145 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:28.145 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:28.145 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:28.145 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:28.145 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:28.146 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:28.146 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:28.146 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:28.146 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:28.147 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:28.147 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:28.147 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:28.147 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:28.147 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:28.148 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:28.148 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:28.148 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:28.148 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:28.149 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:28.149 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:28.149 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:28.149 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:28.149 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:28.150 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:28.150 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:28.150 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:28.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:28.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:28.254 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:28.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:28.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:28.256 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:28.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:28.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:28.257 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:28.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:28.618 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:28.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:28.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:29.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:29.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:29.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:29.101 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:29.101 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:29.102 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:29.699 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T12:58:29.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T12:58:29.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:29.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:29.700 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:29.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:29.793 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:29.872 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:29.872 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:29.873 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:29.873 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:29.873 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:29.873 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:29.874 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:29.874 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:29.874 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:29.874 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:29.875 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:29.875 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:29.875 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:29.875 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:29.875 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:29.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:29.876 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:29.876 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:29.876 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:29.877 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:29.877 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:29.877 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:29.877 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:29.878 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:29.878 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:29.878 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:29.878 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:29.879 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:29.879 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:29.879 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:29.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:29.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:29.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:29.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:29.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:29.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:29.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:29.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:30.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:30.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:30.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:30.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:30.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:30.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:30.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:30.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:30.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:30.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:30.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:30.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:30.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:31.262 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863018 2021-05-12T12:58:31.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863018 '!=' null 2021-05-12T12:58:31.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:31.263 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:31.263 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:31.264 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:31.356 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:31.356 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:31.356 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:31.356 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:31.358 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:31.358 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:31.358 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:31.358 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:31.359 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:31.359 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:31.359 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:31.359 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:31.360 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:31.360 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:31.360 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:31.360 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:31.360 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:31.361 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:31.361 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:31.361 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:31.361 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:31.362 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:31.362 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:31.362 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:31.362 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:31.364 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:31.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:31.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:31.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:31.466 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:31.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:31.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:31.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:31.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:31.824 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:31.824 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:31.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:31.825 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:32.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:32.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:32.305 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:32.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:32.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:32.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:32.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:32.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:32.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:32.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863018 2021-05-12T12:58:32.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863018 '!=' null 2021-05-12T12:58:32.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:32.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:32.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:32.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:32.872 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:32.872 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:32.872 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:32.872 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:32.873 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:32.873 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:32.873 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:32.873 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:32.874 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:32.874 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:32.874 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:32.874 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:32.875 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:32.875 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:32.875 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:32.875 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:32.876 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:32.876 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:32.876 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:32.876 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:32.876 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:32.877 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:32.877 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:32.877 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:32.877 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:32.877 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:32.878 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:32.878 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:32.878 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:32.879 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:32.880 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:32.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:32.982 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:32.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:32.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:32.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:32.983 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:33.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:33.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:33.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:33.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:33.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:33.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:33.858 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:33.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:33.860 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:34.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241964 2021-05-12T12:58:34.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241964 '!=' null 2021-05-12T12:58:34.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:34.351 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:34.352 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:34.352 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:34.444 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:34.444 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:34.444 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:34.444 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:34.445 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:34.445 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:34.445 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:34.445 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:34.445 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:34.446 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:34.446 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:34.446 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:34.446 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:34.447 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:34.447 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:34.447 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:34.447 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:34.448 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:34.448 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:34.448 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:34.448 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:34.448 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:34.449 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:34.449 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:34.449 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:34.449 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:34.450 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:34.450 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:34.450 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:34.450 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:34.452 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:34.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:34.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:34.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:34.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:34.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:34.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:34.556 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:34.557 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:34.908 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:34.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:34.908 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:34.909 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:35.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:35.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:35.428 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:35.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:35.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:35.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:35.429 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:35.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:35.430 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:35.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862899 2021-05-12T12:58:35.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862899 '!=' null 2021-05-12T12:58:35.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:35.907 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:35.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:35.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:35.998 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:35.999 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:36.000 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:36.001 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:36.002 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:36.003 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:36.004 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:36.005 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:36.005 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:36.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:36.107 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:36.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:36.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:36.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:36.109 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:36.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:36.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:36.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:36.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:36.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:36.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:36.460 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:36.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:36.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:36.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:36.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:36.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:36.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:36.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:36.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:36.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:37.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241913 2021-05-12T12:58:37.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241913 '!=' null 2021-05-12T12:58:37.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:37.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:37.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:37.443 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:37.536 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:37.536 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:37.537 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:37.537 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:37.537 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:37.537 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:37.538 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:37.538 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:37.538 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:37.538 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:37.538 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:37.539 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:37.539 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:37.539 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:37.539 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:37.540 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:37.540 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:37.540 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:37.540 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:37.540 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:37.541 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:37.541 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:37.541 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:37.541 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:37.542 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:37.542 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:37.542 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:37.542 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:37.542 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:37.543 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:37.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:37.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:37.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:37.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:37.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:37.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:37.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:37.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:37.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:38.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:38.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:38.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:38.483 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:38.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:38.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:38.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:38.484 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:38.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:38.485 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:38.975 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241913 2021-05-12T12:58:38.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241913 '!=' null 2021-05-12T12:58:38.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:38.976 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:38.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:38.976 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:39.067 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:39.068 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:39.068 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:39.068 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:39.068 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:39.069 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:39.069 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:39.069 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:39.069 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:39.070 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:39.070 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:39.070 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:39.070 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:39.070 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:39.071 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:39.071 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:39.071 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:39.071 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:39.071 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:39.072 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:39.072 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:39.072 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:39.073 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:39.073 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:39.073 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:39.073 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:39.073 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:39.074 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:39.074 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:39.074 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:39.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:39.177 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:39.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:39.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:39.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:39.178 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:39.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:39.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:39.179 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:39.533 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:39.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:39.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:39.534 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:40.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:40.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:40.021 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:40.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:40.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:40.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:40.022 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:40.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:40.023 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:40.509 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241915 2021-05-12T12:58:40.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241915 '!=' null 2021-05-12T12:58:40.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:40.510 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:40.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:40.511 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:40.603 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:40.603 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:40.603 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:40.604 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:40.604 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:40.604 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:40.604 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:40.605 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:40.605 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:40.605 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:40.605 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:40.605 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:40.606 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:40.606 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:40.606 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:40.606 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:40.607 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:40.607 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:40.607 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:40.607 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:40.607 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:40.608 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:40.608 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:40.608 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:40.609 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:40.609 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:40.609 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:40.609 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:40.609 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:40.610 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:40.611 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:40.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:40.713 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:40.713 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:40.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:40.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:40.714 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:40.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:40.715 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:41.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:41.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:41.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:41.070 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:41.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:41.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:41.555 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:41.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:41.556 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:41.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:41.557 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:41.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:41.558 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:42.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862903 2021-05-12T12:58:42.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862903 '!=' null 2021-05-12T12:58:42.039 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:42.040 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:42.040 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:42.040 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:42.134 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:42.135 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:42.135 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:42.135 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:42.135 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:42.136 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:42.136 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:42.136 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:42.136 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:42.137 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:42.137 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:42.137 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:42.137 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:42.137 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:42.138 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:42.138 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:42.138 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:42.138 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:42.139 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:42.139 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:42.139 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:42.139 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:42.139 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:42.140 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:42.140 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:42.140 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:42.140 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:42.140 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:42.141 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:42.141 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:42.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:42.244 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:42.245 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:42.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:42.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:42.245 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:42.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:42.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:42.246 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:42.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:42.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:42.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:42.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:43.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:43.092 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:43.093 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:43.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:43.093 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:43.576 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241914 2021-05-12T12:58:43.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241914 '!=' null 2021-05-12T12:58:43.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:43.578 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:43.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:43.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:43.668 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:43.669 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:43.669 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:43.670 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:43.670 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:43.670 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:43.671 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:43.672 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:43.672 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:43.672 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:43.673 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:43.673 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:43.674 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:43.674 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:43.674 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:43.675 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:43.675 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:43.676 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:43.676 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:43.677 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:43.677 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:43.678 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:43.678 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:43.679 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:43.679 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:43.679 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:43.680 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:43.680 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:43.681 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:43.681 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:43.682 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:43.778 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:43.779 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:43.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:43.779 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:43.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:43.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:43.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:43.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:44.145 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:44.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:44.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:44.146 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:44.622 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:44.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:44.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:44.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:44.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:44.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:44.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:44.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:44.624 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:45.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241914 2021-05-12T12:58:45.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241914 '!=' null 2021-05-12T12:58:45.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:45.103 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:45.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:45.104 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:45.196 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:45.198 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:45.198 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:45.198 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:45.198 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:45.198 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:45.199 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:45.199 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:45.199 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:45.199 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:45.199 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:45.201 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:45.201 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:45.201 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:45.201 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:45.201 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:45.202 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:45.202 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:45.202 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:45.202 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:45.205 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:45.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:45.306 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:45.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:45.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:45.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:45.307 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:45.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:45.308 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:45.664 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:45.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:45.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:45.665 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:46.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:46.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:46.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:46.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:46.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:46.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:46.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:46.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:46.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:46.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862907 2021-05-12T12:58:46.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862907 '!=' null 2021-05-12T12:58:46.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:46.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:46.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:46.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:46.731 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:46.731 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:46.731 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:46.731 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:46.732 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:46.732 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:46.732 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:46.732 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:46.732 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:46.733 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:46.733 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:46.733 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:46.733 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:46.734 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:46.734 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:46.734 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:46.734 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:46.737 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:46.737 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:46.737 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:46.738 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:46.738 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:46.738 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:46.738 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:46.738 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:46.739 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:46.739 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:46.739 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:46.739 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:46.739 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:46.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:46.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:46.841 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:46.841 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:46.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:46.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:46.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:46.842 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:46.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:47.195 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:47.195 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:47.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:47.196 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:47.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:47.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:47.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:47.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:47.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:47.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:47.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:47.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:47.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:48.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241958 2021-05-12T12:58:48.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241958 '!=' null 2021-05-12T12:58:48.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:48.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:48.155 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:48.155 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:48.247 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:48.248 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:48.248 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:48.248 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:48.248 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:48.249 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:48.249 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:48.249 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:48.249 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:48.250 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:48.250 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:48.250 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:48.250 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:48.250 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:48.251 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:48.251 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:48.251 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:48.251 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:48.252 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:48.252 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:48.252 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:48.252 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:48.253 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:48.253 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:48.253 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:48.253 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:48.253 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:48.254 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:48.254 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:48.254 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:48.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:48.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:48.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:48.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:48.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:48.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:48.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:48.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:48.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:48.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:49.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:49.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:49.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:49.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:49.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:49.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:49.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:49.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:49.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:49.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241957 2021-05-12T12:58:49.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241957 '!=' null 2021-05-12T12:58:49.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:49.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:49.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:49.843 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:49.936 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:49.936 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:49.936 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:49.937 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:49.937 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:49.937 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:49.937 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:49.937 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:49.938 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:49.938 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:49.938 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:49.938 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:49.939 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:49.939 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:49.939 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:49.939 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:49.940 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:49.940 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:49.940 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:49.940 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:49.940 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:49.941 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:49.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:49.941 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:49.941 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:49.941 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:49.942 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:49.942 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:49.942 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:49.942 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:49.944 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:50.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:50.046 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:50.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:50.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:50.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:50.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:50.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:50.047 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:50.418 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:50.418 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:50.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:50.419 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:50.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:50.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:50.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:50.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:50.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:50.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:50.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:50.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:50.906 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:51.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862884 2021-05-12T12:58:51.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862884 '!=' null 2021-05-12T12:58:51.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:51.401 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:51.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:51.402 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:51.494 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:51.495 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:51.495 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:51.495 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:51.495 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:51.495 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:51.496 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:51.497 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:51.498 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:51.499 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:51.499 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:51.499 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:51.499 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:51.499 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:51.500 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:51.500 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:51.500 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:51.500 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:51.500 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:51.503 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:51.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:51.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:51.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:51.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:51.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:51.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:51.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:51.606 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:51.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:51.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:51.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:51.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:52.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:52.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:52.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:52.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:52.445 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:52.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:52.446 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:52.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:52.446 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:52.930 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862884 2021-05-12T12:58:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862884 '!=' null 2021-05-12T12:58:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:52.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:52.931 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:52.932 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:53.023 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:53.023 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:53.024 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:53.024 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:53.024 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:53.024 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:53.025 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:53.025 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:53.025 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:53.025 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:53.025 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:53.026 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:53.026 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:53.026 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:53.026 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:53.027 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:53.027 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:53.027 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:53.027 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:53.027 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:53.028 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:53.028 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:53.028 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:53.028 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:53.028 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:53.029 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:53.029 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:53.029 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:53.029 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:53.030 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:53.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:53.133 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:53.134 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:53.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:53.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:53.134 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:53.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:53.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:53.135 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:53.520 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:53.520 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:53.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:53.521 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:54.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:54.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:54.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:54.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:54.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241876 2021-05-12T12:58:54.567 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241876 '!=' null 2021-05-12T12:58:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:54.568 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:54.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:54.568 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:54.662 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:54.662 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:54.663 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:54.663 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:54.663 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:54.663 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:54.663 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:54.664 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:54.664 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:54.664 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:54.664 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:54.665 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:54.665 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:54.665 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:54.665 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:54.665 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:54.666 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:54.666 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:54.666 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:54.666 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:54.667 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:54.667 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:54.667 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:54.667 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:54.667 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:54.668 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:54.668 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:54.668 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:54.668 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:54.668 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:54.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:54.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:54.772 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:54.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:54.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:54.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:54.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:54.773 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:54.774 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:55.131 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:55.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:55.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:55.132 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:55.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:55.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:55.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:55.620 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:55.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:55.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:55.621 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:55.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:55.621 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:56.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862773 2021-05-12T12:58:56.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862773 '!=' null 2021-05-12T12:58:56.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:56.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:56.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:56.114 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:56.207 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:56.208 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:56.208 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:56.208 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:56.210 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:56.210 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:56.210 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:56.210 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:56.211 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:56.211 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:56.211 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:56.211 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:56.211 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:56.212 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:56.212 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:56.212 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:56.213 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:56.213 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:56.213 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:56.213 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:56.213 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:56.214 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:56.214 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:56.214 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:56.214 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:56.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:56.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:56.317 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:56.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:56.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:56.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:56.318 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:56.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:56.319 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:56.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:56.670 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:56.670 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:56.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:57.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:57.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:57.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:57.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:57.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:57.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:57.156 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:57.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:57.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:57.774 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241821 2021-05-12T12:58:57.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241821 '!=' null 2021-05-12T12:58:57.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:57.775 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:57.775 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:57.776 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:57.868 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:57.868 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:57.869 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:57.869 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:57.869 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:57.869 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:57.870 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:57.870 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:57.870 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:57.870 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:57.870 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:57.871 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:57.871 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:57.871 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:57.871 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:57.872 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:57.872 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:57.872 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:57.872 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:57.872 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:57.873 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:57.873 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:57.873 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:57.873 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:57.874 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:57.874 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:57.874 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:57.874 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:57.874 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:57.875 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:57.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:57.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:57.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:57.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:57.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:57.979 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:57.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:57.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:57.980 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:58.353 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:58.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:58.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:58.354 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:58:58.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:58:58.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:58:58.833 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:58:58.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:58:58.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:58:58.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:58:58.834 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:58:58.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:58:58.835 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:58:59.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241821 2021-05-12T12:58:59.327 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241821 '!=' null 2021-05-12T12:58:59.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:58:59.328 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:59.328 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:58:59.328 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:58:59.421 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:58:59.422 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:58:59.422 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:59.422 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:59.422 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:59.423 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:58:59.423 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:59.423 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:58:59.423 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:59.423 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:59.424 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:59.424 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:59.424 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:58:59.424 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:58:59.424 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:59.425 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:59.425 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:58:59.425 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:58:59.425 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:58:59.425 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:58:59.426 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:59.426 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:58:59.426 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:58:59.426 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:58:59.427 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:58:59.427 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:58:59.427 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:58:59.427 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:58:59.427 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:58:59.428 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:58:59.430 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:58:59.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:58:59.531 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:58:59.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:58:59.532 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:58:59.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:58:59.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:58:59.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:58:59.533 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:58:59.904 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:58:59.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:58:59.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:58:59.905 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:00.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:00.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:00.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:00.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:00.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:00.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:00.389 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:00.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:00.389 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:00.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241802 2021-05-12T12:59:00.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241802 '!=' null 2021-05-12T12:59:00.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:00.893 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:00.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:00.894 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:00.985 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:00.986 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:00.987 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:00.988 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:00.989 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:00.990 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:00.991 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:00.991 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:00.991 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:00.991 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:00.991 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:00.992 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:00.992 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:00.993 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:01.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:01.095 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:01.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:01.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:01.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:01.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:01.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:01.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:01.455 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:01.455 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:01.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:01.456 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:02.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:02.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:02.044 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:02.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:02.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:02.046 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:02.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989034 2021-05-12T12:59:02.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989034 '!=' null 2021-05-12T12:59:02.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:02.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:02.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:02.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:02.619 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:02.619 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:02.619 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:02.620 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:02.620 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:02.620 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:02.620 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:02.621 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:02.621 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:02.621 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:02.621 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:02.621 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:02.622 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:02.622 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:02.622 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:02.622 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:02.622 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:02.623 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:02.623 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:02.623 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:02.623 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:02.623 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:02.624 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:02.624 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:02.624 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:02.624 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:02.624 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:02.625 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:02.625 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:02.625 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:02.627 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:02.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:02.728 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:02.729 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:02.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:02.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:02.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:02.730 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:02.731 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:03.096 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:03.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:03.096 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:03.097 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:03.614 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:03.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:03.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:03.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:03.615 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:03.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:03.616 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:03.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:03.616 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:04.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368132 2021-05-12T12:59:04.099 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368132 '!=' null 2021-05-12T12:59:04.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:04.100 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:04.100 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:04.100 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:04.193 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:04.193 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:04.194 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:04.194 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:04.194 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:04.194 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:04.194 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:04.195 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:04.195 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:04.195 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:04.195 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:04.196 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:04.196 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:04.196 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:04.196 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:04.197 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:04.197 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:04.197 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:04.197 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:04.197 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:04.198 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:04.198 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:04.198 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:04.198 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:04.199 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:04.199 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:04.199 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:04.199 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:04.200 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:04.200 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:04.201 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:04.303 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:04.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:04.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:04.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:04.304 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:04.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:04.305 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:04.663 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:04.663 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:04.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:04.664 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:05.142 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:05.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:05.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:05.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:05.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:05.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:05.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:05.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:05.144 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:05.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6988999 2021-05-12T12:59:05.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6988999 '!=' null 2021-05-12T12:59:05.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:05.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:05.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:05.642 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:05.734 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:05.734 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:05.734 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:05.735 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:05.735 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:05.735 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:05.735 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:05.735 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:05.736 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:05.736 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:05.736 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:05.736 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:05.736 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:05.737 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:05.737 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:05.737 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:05.737 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:05.737 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:05.738 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:05.738 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:05.738 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:05.738 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:05.739 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:05.739 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:05.739 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:05.739 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:05.739 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:05.740 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:05.740 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:05.740 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:05.742 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:05.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:05.844 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:05.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:05.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:05.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:05.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:05.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:05.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:06.203 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:06.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:06.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:06.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:06.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:06.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:06.688 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:06.689 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:06.689 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:06.690 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:07.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6988999 2021-05-12T12:59:07.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6988999 '!=' null 2021-05-12T12:59:07.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:07.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:07.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:07.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:07.270 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:07.270 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:07.270 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:07.270 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:07.271 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:07.271 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:07.271 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:07.271 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:07.272 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:07.272 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:07.272 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:07.272 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:07.272 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:07.273 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:07.273 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:07.273 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:07.273 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:07.274 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:07.274 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:07.274 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:07.274 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:07.274 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:07.275 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:07.275 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:07.275 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:07.275 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:07.276 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:07.276 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:07.276 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:07.276 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:07.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:07.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:07.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:07.380 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:07.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:07.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:07.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:07.381 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:07.382 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:07.736 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:07.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:07.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:07.737 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:08.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:08.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:08.222 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:08.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:08.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:08.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:08.223 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:08.223 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:08.224 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:08.707 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368159 2021-05-12T12:59:08.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368159 '!=' null 2021-05-12T12:59:08.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:08.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:08.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:08.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:08.802 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:08.802 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:08.802 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:08.802 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:08.803 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:08.803 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:08.803 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:08.803 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:08.804 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:08.804 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:08.804 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:08.804 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:08.805 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:08.805 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:08.805 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:08.805 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:08.805 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:08.806 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:08.806 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:08.806 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:08.806 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:08.807 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:08.807 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:08.807 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:08.807 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:08.807 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:08.808 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:08.808 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:08.808 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:08.808 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:08.810 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:08.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:08.912 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:08.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:08.912 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:08.913 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:09.269 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:09.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:09.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:09.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:09.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:09.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:09.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:09.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:09.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:09.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:09.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:09.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:09.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:10.235 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368197 2021-05-12T12:59:10.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368197 '!=' null 2021-05-12T12:59:10.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:10.236 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:10.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:10.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:10.329 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:10.330 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:10.330 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:10.330 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:10.330 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:10.331 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:10.331 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:10.331 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:10.331 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:10.332 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:10.332 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:10.332 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:10.332 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:10.332 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:10.333 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:10.333 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:10.333 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:10.333 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:10.334 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:10.334 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:10.334 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:10.334 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:10.334 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:10.335 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:10.335 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:10.335 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:10.335 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:10.335 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:10.336 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:10.336 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:10.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:10.439 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:10.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:10.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:10.440 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:10.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:10.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:10.441 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:10.791 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:10.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:10.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:10.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:11.274 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:11.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:11.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:11.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:11.275 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:11.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:11.276 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:11.276 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:11.276 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:11.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989151 2021-05-12T12:59:11.829 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989151 '!=' null 2021-05-12T12:59:11.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:11.830 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:11.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:11.830 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:11.923 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:11.923 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:11.924 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:11.925 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:11.926 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:11.927 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:11.928 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:11.929 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:11.929 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:11.931 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:12.032 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:12.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:12.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:12.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:12.033 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:12.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:12.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:12.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:12.392 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:12.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:12.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:12.393 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:12.875 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:12.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:12.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:12.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:12.876 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:12.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:12.877 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:12.877 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:12.877 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:13.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989151 2021-05-12T12:59:13.358 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989151 '!=' null 2021-05-12T12:59:13.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:13.359 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:13.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:13.359 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:13.454 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:13.454 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:13.454 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:13.455 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:13.455 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:13.455 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:13.455 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:13.456 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:13.456 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:13.456 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:13.456 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:13.456 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:13.457 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:13.457 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:13.457 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:13.457 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:13.458 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:13.458 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:13.458 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:13.458 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:13.458 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:13.459 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:13.459 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:13.459 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:13.459 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:13.459 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:13.460 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:13.460 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:13.460 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:13.460 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:13.462 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:13.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:13.564 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:13.564 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:13.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:13.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:13.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:13.565 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:13.566 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:13.943 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:13.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:13.944 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:13.945 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:14.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:14.420 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:14.421 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:14.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:14.422 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:14.905 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241847 2021-05-12T12:59:14.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241847 '!=' null 2021-05-12T12:59:14.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:14.906 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:14.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:14.907 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:15.001 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:15.001 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:15.002 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:15.002 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:15.002 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:15.002 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:15.003 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:15.003 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:15.003 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:15.003 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:15.003 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:15.004 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:15.004 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:15.004 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:15.004 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:15.005 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:15.005 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:15.005 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:15.005 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:15.005 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:15.006 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:15.006 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:15.006 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:15.006 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:15.007 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:15.007 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:15.007 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:15.007 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:15.008 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:15.008 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:15.009 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:15.111 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:15.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:15.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:15.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:15.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:15.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:15.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:15.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:15.476 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:15.476 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:15.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:15.477 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:15.959 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:15.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:15.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:15.960 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:15.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:15.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:15.961 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:15.961 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:15.962 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:16.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862731 2021-05-12T12:59:16.456 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862731 '!=' null 2021-05-12T12:59:16.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:16.457 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:16.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:16.457 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:16.550 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:16.550 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:16.550 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:16.550 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:16.551 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:16.551 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:16.551 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:16.551 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:16.551 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:16.552 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:16.552 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:16.552 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:16.553 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:16.553 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:16.553 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:16.553 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:16.553 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:16.554 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:16.554 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:16.554 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:16.554 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:16.554 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:16.555 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:16.555 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:16.555 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:16.555 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:16.556 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:16.556 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:16.556 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:16.556 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:16.558 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:16.659 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:16.660 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:16.661 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:16.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:16.662 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:17.043 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:17.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:17.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:17.044 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:17.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:17.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:17.525 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:17.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:17.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:17.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:17.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:17.526 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:17.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:18.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241858 2021-05-12T12:59:18.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241858 '!=' null 2021-05-12T12:59:18.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:18.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:18.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:18.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:18.101 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:18.102 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:18.102 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:18.103 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:18.103 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:18.104 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:18.104 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:18.104 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:18.105 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:18.105 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:18.106 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:18.106 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:18.107 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:18.107 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:18.108 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:18.108 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:18.108 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:18.109 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:18.109 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:18.110 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:18.110 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:18.111 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:18.111 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:18.112 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:18.112 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:18.112 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:18.113 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:18.113 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:18.114 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:18.114 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:18.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:18.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:18.212 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:18.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:18.213 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:18.213 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:18.213 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:18.213 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:18.214 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:18.577 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:18.577 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:18.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:18.578 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:19.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:19.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:19.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:19.064 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:19.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:19.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:19.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:19.065 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:19.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:19.551 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241814 2021-05-12T12:59:19.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241814 '!=' null 2021-05-12T12:59:19.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:19.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:19.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:19.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:19.645 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:19.645 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:19.645 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:19.645 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:19.646 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:19.646 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:19.646 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:19.646 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:19.646 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:19.647 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:19.647 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:19.647 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:19.647 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:19.648 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:19.648 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:19.648 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:19.648 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:19.648 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:19.649 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:19.649 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:19.649 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:19.649 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:19.649 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:19.650 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:19.650 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:19.650 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:19.650 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:19.650 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:19.651 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:19.651 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:19.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:19.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:19.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:19.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:19.755 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:19.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:19.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:19.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:19.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:20.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:20.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:20.111 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:20.111 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:20.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:20.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:20.599 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:20.600 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:20.600 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:20.601 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:21.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241814 2021-05-12T12:59:21.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241814 '!=' null 2021-05-12T12:59:21.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:21.077 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:21.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:21.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:21.170 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:21.170 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:21.170 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:21.171 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:21.171 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:21.171 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:21.171 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:21.171 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:21.172 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:21.173 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:21.174 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:21.175 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:21.175 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:21.175 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:21.175 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:21.176 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:21.176 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:21.176 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:21.176 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:21.176 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:21.178 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:21.280 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:21.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:21.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:21.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:21.281 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:21.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:21.282 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:21.644 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:21.644 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:21.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:21.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:22.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:22.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:22.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:22.129 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:22.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:22.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:22.130 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:22.130 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:22.131 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:22.609 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862660 2021-05-12T12:59:22.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862660 '!=' null 2021-05-12T12:59:22.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:22.610 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:22.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:22.611 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:22.703 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:22.704 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:22.704 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:22.704 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:22.705 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:22.705 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:22.705 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:22.705 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:22.705 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:22.706 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:22.706 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:22.706 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:22.706 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:22.707 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:22.707 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:22.707 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:22.707 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:22.707 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:22.708 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:22.708 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:22.708 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:22.708 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:22.709 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:22.709 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:22.709 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:22.709 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:22.709 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:22.710 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:22.710 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:22.710 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:22.712 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:22.813 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:22.814 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:22.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:22.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:22.814 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:22.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:22.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:22.815 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:23.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:23.173 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:23.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:23.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:23.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:23.754 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:23.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:23.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:23.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:23.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:23.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:23.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:23.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:24.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241738 2021-05-12T12:59:24.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241738 '!=' null 2021-05-12T12:59:24.239 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:24.240 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:24.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:24.240 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:24.333 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:24.334 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:24.334 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:24.334 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:24.334 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:24.335 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:24.335 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:24.335 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:24.335 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:24.336 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:24.336 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:24.336 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:24.336 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:24.336 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:24.337 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:24.337 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:24.337 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:24.337 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:24.338 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:24.338 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:24.338 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:24.338 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:24.339 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:24.339 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:24.339 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:24.339 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:24.340 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:24.340 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:24.340 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:24.340 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:24.341 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:24.443 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:24.444 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:24.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:24.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:24.444 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:24.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:24.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:24.445 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:24.800 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:24.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:24.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:24.801 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:25.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:25.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:25.288 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:25.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:25.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:25.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:25.289 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:25.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:25.290 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:25.842 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862631 2021-05-12T12:59:25.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862631 '!=' null 2021-05-12T12:59:25.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:25.843 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:25.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:25.844 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:25.937 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:25.938 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:25.938 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:25.938 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:25.939 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:25.939 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:25.939 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:25.939 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:25.939 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:25.940 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:25.940 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:25.940 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:25.940 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:25.941 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:25.941 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:25.941 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:25.941 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:25.942 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:25.942 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:25.942 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:25.942 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:25.943 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:25.943 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:25.943 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:25.943 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:25.943 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:25.944 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:25.944 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:25.944 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:25.944 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:25.946 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:26.048 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:26.049 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:26.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:26.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:26.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:26.049 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:26.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:26.050 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:26.417 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:26.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:26.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:26.417 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:26.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:26.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:26.902 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:26.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:26.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:26.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:26.903 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:26.903 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:26.904 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:27.399 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241836 2021-05-12T12:59:27.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241836 '!=' null 2021-05-12T12:59:27.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:27.400 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:27.400 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:27.401 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:27.492 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:27.493 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:27.493 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:27.493 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:27.493 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:27.494 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:27.494 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:27.494 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:27.494 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:27.495 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:27.495 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:27.495 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:27.495 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:27.496 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:27.496 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:27.496 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:27.496 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:27.497 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:27.497 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:27.497 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:27.497 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:27.498 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:27.498 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:27.498 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:27.499 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:27.499 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:27.499 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:27.500 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:27.500 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:27.500 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:27.501 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:27.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:27.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:27.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:27.603 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:27.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:27.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:27.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:27.604 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:27.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:27.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:27.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:27.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:28.459 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:28.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:28.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:28.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:28.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:28.460 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:28.461 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:28.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:28.461 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:28.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241836 2021-05-12T12:59:28.941 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241836 '!=' null 2021-05-12T12:59:28.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:28.942 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:28.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:28.943 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:29.037 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:29.037 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:29.037 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:29.037 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:29.038 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:29.038 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:29.038 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:29.038 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:29.039 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:29.039 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:29.039 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:29.039 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:29.039 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:29.040 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:29.040 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:29.040 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:29.040 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:29.041 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:29.041 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:29.041 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:29.041 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:29.042 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:29.042 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:29.042 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:29.042 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:29.043 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:29.043 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:29.043 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:29.043 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:29.043 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:29.045 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:29.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:29.146 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:29.147 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:29.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:29.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:29.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:29.148 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:29.149 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:29.508 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:29.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:29.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:29.509 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:30.004 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:30.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:30.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:30.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:30.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:30.005 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:30.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:30.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:30.006 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:30.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241831 2021-05-12T12:59:30.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241831 '!=' null 2021-05-12T12:59:30.485 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:30.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:30.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:30.486 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:30.580 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:30.581 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:30.581 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:30.581 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:30.581 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:30.582 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:30.582 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:30.582 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:30.582 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:30.583 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:30.583 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:30.583 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:30.583 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:30.584 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:30.584 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:30.584 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:30.584 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:30.585 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:30.585 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:30.585 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:30.585 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:30.586 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:30.586 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:30.586 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:30.586 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:30.587 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:30.587 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:30.587 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:30.587 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:30.587 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:30.589 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:30.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:30.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:30.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:30.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:30.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:30.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:30.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:30.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:31.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:31.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:31.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:31.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:31.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:31.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:31.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:31.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:31.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:31.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:31.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:31.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:31.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:32.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862829 2021-05-12T12:59:32.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862829 '!=' null 2021-05-12T12:59:32.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:32.008 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:32.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:32.009 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:32.100 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:32.100 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:32.100 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:32.101 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:32.101 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:32.101 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:32.101 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:32.102 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:32.102 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:32.102 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:32.102 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:32.103 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:32.103 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:32.103 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:32.103 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:32.103 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:32.104 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:32.104 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:32.104 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:32.104 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:32.105 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:32.105 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:32.105 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:32.105 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:32.105 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:32.106 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:32.106 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:32.106 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:32.106 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:32.107 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:32.108 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:32.209 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:32.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:32.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:32.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:32.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:32.571 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:32.571 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:32.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:32.572 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:33.055 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:33.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:33.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:33.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:33.056 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:33.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:33.057 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:33.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:33.057 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:33.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241892 2021-05-12T12:59:33.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241892 '!=' null 2021-05-12T12:59:33.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:33.544 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:33.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:33.544 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:33.637 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:33.637 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:33.637 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:33.638 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:33.638 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:33.638 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:33.638 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:33.639 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:33.639 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:33.639 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:33.639 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:33.639 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:33.640 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:33.640 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:33.640 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:33.640 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:33.641 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:33.641 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:33.641 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:33.641 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:33.641 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:33.642 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:33.642 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:33.642 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:33.642 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:33.642 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:33.643 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:33.643 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:33.643 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:33.643 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:33.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:33.747 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:33.748 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:33.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:33.748 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:33.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:33.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:33.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:33.749 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:34.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:34.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:34.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:34.110 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:34.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:34.603 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:34.604 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:34.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:34.605 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:35.097 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241892 2021-05-12T12:59:35.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241892 '!=' null 2021-05-12T12:59:35.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:35.098 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:35.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:35.099 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:35.191 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:35.192 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:35.192 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:35.192 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:35.192 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:35.193 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:35.193 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:35.193 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:35.193 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:35.193 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:35.194 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:35.194 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:35.194 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:35.194 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:35.196 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:35.196 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:35.196 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:35.196 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:35.196 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:35.197 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:35.197 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:35.197 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:35.197 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:35.198 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:35.198 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:35.199 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:35.301 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:35.302 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:35.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:35.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:35.302 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:35.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:35.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:35.303 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:35.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:35.677 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:35.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:35.678 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:36.161 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:36.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:36.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:36.162 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:36.163 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:36.163 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:36.164 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:36.640 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862904 2021-05-12T12:59:36.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862904 '!=' null 2021-05-12T12:59:36.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:36.641 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:36.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:36.641 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:36.736 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:36.736 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:36.736 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:36.736 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:36.737 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:36.737 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:36.737 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:36.737 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:36.737 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:36.738 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:36.738 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:36.738 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:36.738 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:36.739 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:36.739 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:36.739 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:36.739 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:36.739 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:36.740 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:36.740 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:36.740 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:36.740 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:36.740 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:36.741 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:36.741 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:36.741 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:36.741 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:36.741 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:36.742 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:36.742 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:36.744 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:36.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:36.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:36.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:36.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:36.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:36.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:36.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:36.848 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:37.206 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:37.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:37.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:37.207 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:37.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:37.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:37.739 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:37.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:37.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:37.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:37.740 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:37.740 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:37.741 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:38.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241969 2021-05-12T12:59:38.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241969 '!=' null 2021-05-12T12:59:38.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:38.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:38.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:38.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:38.326 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:38.327 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:38.327 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:38.327 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:38.327 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:38.328 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:38.328 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:38.328 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:38.328 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:38.329 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:38.329 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:38.329 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:38.329 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:38.329 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:38.330 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:38.330 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:38.330 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:38.330 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:38.331 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:38.331 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:38.331 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:38.331 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:38.331 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:38.332 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:38.332 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:38.332 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:38.332 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:38.332 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:38.333 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:38.333 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:38.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:38.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:38.437 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:38.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:38.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:38.437 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:38.438 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:38.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:38.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:38.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:38.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:39.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:39.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:39.292 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:39.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:39.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:39.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:39.293 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:39.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:39.294 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:39.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T12:59:39.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T12:59:39.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:39.796 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:39.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:39.797 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:39.890 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:39.890 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:39.890 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:39.891 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:39.891 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:39.891 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:39.891 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:39.892 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:39.892 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:39.892 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:39.892 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:39.892 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:39.893 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:39.893 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:39.893 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:39.893 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:39.894 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:39.894 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:39.894 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:39.894 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:39.894 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:39.895 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:39.895 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:39.895 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:39.895 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:39.895 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:39.896 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:39.896 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:39.896 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:39.896 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:39.898 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:40.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:40.000 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:40.000 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:40.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:40.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:40.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:40.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:40.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:40.355 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:40.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:40.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:40.356 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:40.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:40.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:40.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:40.846 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:40.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:40.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:40.847 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:40.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:40.848 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:41.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862864 2021-05-12T12:59:41.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862864 '!=' null 2021-05-12T12:59:41.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:41.432 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:41.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:41.433 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:41.526 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:41.526 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:41.526 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:41.527 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:41.527 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:41.527 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:41.527 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:41.527 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:41.528 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:41.528 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:41.528 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:41.528 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:41.529 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:41.529 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:41.529 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:41.529 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:41.529 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:41.530 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:41.530 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:41.530 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:41.530 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:41.531 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:41.531 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:41.531 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:41.531 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:41.532 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:41.532 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:41.532 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:41.532 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:41.532 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:41.534 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:41.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:41.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:41.636 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:41.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:41.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:41.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:41.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:41.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:41.991 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:41.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:41.991 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:41.992 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:42.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:42.465 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:42.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:42.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:42.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:42.466 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:42.467 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:42.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:42.467 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:42.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862864 2021-05-12T12:59:42.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862864 '!=' null 2021-05-12T12:59:42.977 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:42.978 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:42.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:42.978 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:43.070 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:43.070 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:43.071 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:43.071 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:43.071 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:43.071 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:43.072 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:43.072 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:43.072 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:43.072 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:43.072 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:43.073 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:43.073 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:43.073 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:43.073 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:43.074 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:43.074 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:43.074 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:43.074 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:43.075 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:43.075 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:43.075 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:43.075 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:43.075 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:43.076 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:43.076 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:43.076 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:43.076 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:43.076 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:43.077 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:43.078 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:43.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:43.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:43.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:43.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:43.181 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:43.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:43.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:43.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:43.540 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:43.540 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:43.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:43.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:44.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:44.025 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:44.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:44.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:44.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:44.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:44.026 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:44.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:44.027 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:44.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241878 2021-05-12T12:59:44.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241878 '!=' null 2021-05-12T12:59:44.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:44.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:44.554 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:44.554 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:44.647 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:44.648 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:44.648 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:44.648 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:44.648 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:44.649 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:44.649 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:44.649 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:44.649 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:44.649 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:44.650 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:44.650 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:44.650 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:44.651 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:44.651 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:44.651 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:44.651 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:44.651 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:44.652 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:44.652 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:44.652 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:44.653 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:44.653 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:44.653 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:44.653 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:44.653 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:44.654 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:44.654 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:44.654 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:44.655 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:44.655 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:44.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:44.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:44.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:44.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:44.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:44.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:44.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:44.760 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:45.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:45.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:45.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:45.116 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:45.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:45.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:45.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:45.633 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:45.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:45.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:45.634 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:45.634 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:45.635 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:46.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989206 2021-05-12T12:59:46.121 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989206 '!=' null 2021-05-12T12:59:46.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:46.122 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:46.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:46.122 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:46.213 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:46.215 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:46.215 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:46.215 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:46.215 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:46.216 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:46.216 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:46.216 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:46.216 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:46.217 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:46.217 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:46.217 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:46.217 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:46.218 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:46.218 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:46.218 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:46.218 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:46.218 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:46.219 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:46.219 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:46.219 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:46.219 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:46.220 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:46.220 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:46.220 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:46.220 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:46.221 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:46.323 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:46.324 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:46.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:46.324 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:46.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:46.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:46.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:46.325 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:46.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:46.679 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:46.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:46.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:47.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:47.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:47.175 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:47.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:47.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:47.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:47.176 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:47.176 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:47.177 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:47.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368241 2021-05-12T12:59:47.657 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368241 '!=' null 2021-05-12T12:59:47.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:47.658 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:47.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:47.658 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:47.751 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:47.752 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:47.752 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:47.752 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:47.753 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:47.753 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:47.753 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:47.753 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:47.754 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:47.754 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:47.754 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:47.754 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:47.755 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:47.755 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:47.755 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:47.755 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:47.755 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:47.756 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:47.756 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:47.756 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:47.756 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:47.757 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:47.757 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:47.757 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:47.757 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:47.757 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:47.758 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:47.758 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:47.758 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:47.758 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:47.760 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:47.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:47.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:47.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:47.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:47.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:47.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:47.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:47.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:48.230 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:48.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:48.231 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:48.232 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:48.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:48.721 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:48.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:48.722 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:48.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:48.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:49.207 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368241 2021-05-12T12:59:49.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368241 '!=' null 2021-05-12T12:59:49.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:49.208 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:49.208 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:49.209 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:49.301 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:49.302 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:49.303 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:49.304 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:49.305 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:49.306 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:49.307 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:49.308 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:49.308 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:49.309 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:49.411 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:49.412 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:49.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:49.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:49.412 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:49.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:49.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:49.413 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:49.780 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:49.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:49.780 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:49.781 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:50.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:50.282 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:50.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:50.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:50.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:50.283 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:50.284 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:50.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:50.284 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:50.798 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368245 2021-05-12T12:59:50.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368245 '!=' null 2021-05-12T12:59:50.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:50.799 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:50.799 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:50.800 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:50.892 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:50.892 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:50.893 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:50.893 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:50.893 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:50.893 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:50.894 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:50.894 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:50.894 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:50.894 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:50.894 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:50.895 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:50.895 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:50.895 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:50.895 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:50.896 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:50.896 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:50.896 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:50.896 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:50.896 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:50.897 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:50.897 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:50.897 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:50.897 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:50.898 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:50.898 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:50.898 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:50.898 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:50.899 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:50.899 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:50.900 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:51.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:51.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:51.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:51.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:51.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:51.003 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:51.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:51.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:51.360 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:51.360 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:51.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:51.361 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:51.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:51.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:51.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:51.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:51.850 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:51.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:51.851 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:51.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:51.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:52.334 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989245 2021-05-12T12:59:52.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989245 '!=' null 2021-05-12T12:59:52.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:52.335 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:52.335 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:52.336 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:52.428 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:52.428 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:52.429 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:52.429 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:52.429 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:52.429 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:52.430 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:52.430 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:52.430 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:52.430 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:52.430 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:52.431 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:52.431 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:52.431 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:52.431 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:52.432 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:52.432 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:52.432 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:52.432 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:52.432 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:52.433 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:52.433 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:52.433 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:52.433 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:52.434 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:52.434 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:52.434 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:52.434 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:52.434 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:52.435 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:52.436 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:52.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:52.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:52.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:52.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:52.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:52.541 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:52.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:52.542 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:52.892 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:52.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:52.892 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:52.893 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:53.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:53.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:53.512 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:53.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:53.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:53.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:53.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:53.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:53.514 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:54.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368301 2021-05-12T12:59:54.011 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368301 '!=' null 2021-05-12T12:59:54.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:54.012 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:54.012 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:54.013 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:54.105 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:54.105 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:54.106 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:54.106 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:54.106 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:54.106 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:54.107 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:54.107 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:54.107 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:54.107 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:54.108 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:54.108 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:54.108 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:54.108 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:54.109 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:54.109 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:54.109 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:54.109 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:54.109 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:54.110 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:54.110 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:54.110 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:54.110 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:54.111 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:54.111 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:54.111 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:54.111 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:54.111 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:54.112 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:54.112 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:54.113 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:54.215 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:54.216 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:54.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:54.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:54.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:54.216 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:54.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:54.217 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:54.573 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:54.573 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:54.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:54.574 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:55.059 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:55.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:55.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:55.060 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:55.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:55.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:55.061 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:55.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:55.061 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:55.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T12:59:55.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T12:59:55.548 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:55.549 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:55.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:55.549 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:55.641 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:55.642 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:55.642 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:55.642 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:55.642 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:55.643 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:55.643 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:55.643 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:55.643 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:55.643 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:55.644 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:55.644 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:55.644 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:55.644 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:55.644 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:55.645 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:55.645 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:55.645 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:55.645 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:55.646 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:55.646 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:55.646 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:55.646 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:55.646 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:55.647 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:55.647 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:55.647 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:55.647 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:55.648 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:55.648 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:55.649 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:55.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:55.752 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:55.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:55.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:55.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:55.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:55.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:55.753 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:56.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:56.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:56.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:56.103 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:56.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:56.586 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:56.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:56.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:56.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:56.587 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:56.588 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:56.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:56.588 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:57.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T12:59:57.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T12:59:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:57.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:57.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:57.066 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:57.158 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:57.158 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:57.159 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:57.159 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:57.159 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:57.159 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:57.159 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:57.160 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:57.160 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:57.160 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:57.160 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:57.161 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:57.161 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:57.161 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:57.161 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:57.162 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:57.162 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:57.162 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:57.162 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:57.162 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:57.163 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:57.163 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:57.163 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:57.163 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:57.164 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:57.164 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:57.164 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:57.164 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:57.165 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:57.165 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:57.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:57.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:57.268 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:57.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:57.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:57.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:57.269 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:57.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:57.270 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:57.619 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:57.619 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:57.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:57.620 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:58.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:58.089 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:58.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:58.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:58.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:58.090 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:58.091 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:58.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:58.091 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T12:59:58.581 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241989 2021-05-12T12:59:58.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241989 '!=' null 2021-05-12T12:59:58.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T12:59:58.582 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:58.582 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T12:59:58.583 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T12:59:58.673 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T12:59:58.674 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:58.674 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:58.674 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:58.674 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T12:59:58.675 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:58.675 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T12:59:58.675 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:58.675 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:58.675 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:58.676 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:58.676 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T12:59:58.676 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T12:59:58.677 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:58.677 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:58.677 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T12:59:58.678 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T12:59:58.678 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T12:59:58.678 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T12:59:58.678 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:58.678 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T12:59:58.679 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T12:59:58.679 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T12:59:58.679 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T12:59:58.679 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T12:59:58.680 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T12:59:58.680 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T12:59:58.680 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T12:59:58.680 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T12:59:58.681 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T12:59:58.782 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T12:59:58.783 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T12:59:58.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T12:59:58.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T12:59:58.784 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T12:59:58.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T12:59:58.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T12:59:58.785 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T12:59:59.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T12:59:59.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T12:59:59.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T12:59:59.142 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T12:59:59.677 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T12:59:59.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T12:59:59.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T12:59:59.678 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T12:59:59.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T12:59:59.679 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T12:59:59.680 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T12:59:59.680 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T12:59:59.681 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:00.153 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241999 2021-05-12T13:00:00.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241999 '!=' null 2021-05-12T13:00:00.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:00.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:00.154 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:00.155 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:00.246 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:00.246 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:00.246 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:00.247 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:00.247 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:00.247 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:00.248 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:00.248 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:00.248 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:00.248 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:00.248 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:00.249 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:00.249 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:00.249 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:00.249 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:00.250 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:00.250 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:00.250 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:00.250 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:00.251 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:00.251 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:00.251 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:00.251 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:00.252 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:00.252 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:00.252 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:00.252 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:00.253 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:00.253 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:00.253 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:00.255 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:00.356 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:00.357 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:00.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:00.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:00.357 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:00.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:00.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:00.358 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:00.708 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:00.708 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:00.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:00.709 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:01.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:01.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:01.183 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:01.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:01.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:01.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:01.184 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:01.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:01.185 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:01.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862978 2021-05-12T13:00:01.951 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862978 '!=' null 2021-05-12T13:00:01.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:01.952 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:01.952 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:01.953 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:02.045 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:02.045 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:02.045 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:02.046 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:02.046 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:02.046 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:02.046 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:02.047 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:02.047 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:02.047 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:02.047 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:02.048 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:02.048 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:02.048 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:02.048 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:02.048 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:02.049 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:02.049 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:02.049 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:02.049 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:02.050 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:02.050 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:02.050 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:02.050 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:02.051 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:02.051 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:02.051 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:02.051 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:02.051 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:02.052 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:02.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:02.154 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:02.155 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:02.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:02.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:02.156 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:02.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:02.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:02.157 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:02.507 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:02.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:02.507 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:02.508 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:03.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:03.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:03.002 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:03.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:03.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:03.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:03.003 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:03.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:03.004 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:03.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241924 2021-05-12T13:00:03.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241924 '!=' null 2021-05-12T13:00:03.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:03.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:03.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:03.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:03.579 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:03.580 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:03.580 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:03.580 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:03.581 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:03.581 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:03.581 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:03.582 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:03.582 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:03.582 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:03.583 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:03.583 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:03.583 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:03.584 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:03.584 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:03.585 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:03.585 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:03.585 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:03.586 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:03.586 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:03.586 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:03.587 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:03.587 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:03.587 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:03.588 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:03.588 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:03.588 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:03.589 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:03.589 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:03.589 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:03.590 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:03.690 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:03.691 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:03.691 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:03.692 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:03.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:03.693 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:04.053 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:04.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:04.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:04.054 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:04.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:04.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:04.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:04.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:04.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:04.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:04.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:04.530 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:04.530 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:04.997 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241924 2021-05-12T13:00:04.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241924 '!=' null 2021-05-12T13:00:04.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:04.998 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:04.998 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:04.999 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:05.093 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:05.094 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:05.094 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:05.094 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:05.095 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:05.095 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:05.095 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:05.095 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:05.095 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:05.096 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:05.096 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:05.096 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:05.096 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:05.097 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:05.097 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:05.097 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:05.097 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:05.098 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:05.098 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:05.098 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:05.098 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:05.099 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:05.099 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:05.099 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:05.099 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:05.100 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:05.100 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:05.100 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:05.100 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:05.100 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:05.102 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:05.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:05.204 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:05.204 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:05.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:05.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:05.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:05.205 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:05.206 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:05.554 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:05.554 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:05.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:05.555 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:06.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:06.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:06.075 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:06.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:06.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:06.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:06.076 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:06.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:06.077 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:06.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862877 2021-05-12T13:00:06.552 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862877 '!=' null 2021-05-12T13:00:06.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:06.553 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:06.553 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:06.554 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:06.645 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:06.645 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:06.645 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:06.645 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:06.646 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:06.646 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:06.646 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:06.647 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:06.647 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:06.647 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:06.647 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:06.648 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:06.648 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:06.648 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:06.648 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:06.649 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:06.649 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:06.649 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:06.649 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:06.649 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:06.650 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:06.650 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:06.650 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:06.651 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:06.651 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:06.651 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:06.651 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:06.652 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:06.652 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:06.652 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:06.653 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:06.755 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:06.756 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:06.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:06.756 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:06.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:06.757 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:06.758 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:06.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:07.112 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:07.112 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:07.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:07.113 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:07.623 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:07.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:07.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:07.624 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:07.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:07.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:07.625 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:07.625 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:07.626 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:08.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241902 2021-05-12T13:00:08.114 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241902 '!=' null 2021-05-12T13:00:08.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:08.115 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:08.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:08.115 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:08.208 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:08.208 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:08.209 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:08.209 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:08.210 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:08.210 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:08.210 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:08.211 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:08.211 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:08.211 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:08.212 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:08.212 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:08.212 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:08.212 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:08.213 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:08.213 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:08.213 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:08.213 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:08.214 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:08.214 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:08.214 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:08.214 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:08.215 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:08.215 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:08.215 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:08.215 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:08.216 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:08.216 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:08.216 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:08.217 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:08.217 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:08.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:08.319 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:08.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:08.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:08.320 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:08.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:08.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:08.321 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:08.670 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:08.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:08.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:08.671 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:09.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:09.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:09.172 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:09.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:09.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:09.173 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:09.174 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:09.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:09.174 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:09.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T13:00:09.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T13:00:09.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:09.646 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:09.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:09.647 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:09.738 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:09.738 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:09.739 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:09.739 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:09.739 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:09.739 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:09.739 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:09.740 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:09.740 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:09.740 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:09.741 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:09.741 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:09.741 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:09.741 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:09.742 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:09.742 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:09.742 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:09.742 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:09.742 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:09.743 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:09.743 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:09.743 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:09.743 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:09.744 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:09.744 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:09.744 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:09.744 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:09.744 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:09.745 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:09.745 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:09.746 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:09.848 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:09.849 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:09.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:09.850 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:09.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:09.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:09.851 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:09.852 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:10.200 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:10.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:10.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:10.200 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:10.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:10.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:10.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:10.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:10.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:10.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:10.701 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:10.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:10.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:11.180 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T13:00:11.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T13:00:11.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:11.181 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:11.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:11.182 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:11.276 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:11.276 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:11.276 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:11.277 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:11.277 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:11.277 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:11.277 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:11.278 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:11.278 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:11.278 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:11.279 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:11.279 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:11.279 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:11.280 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:11.280 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:11.281 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:11.281 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:11.281 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:11.282 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:11.282 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:11.283 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:11.283 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:11.283 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:11.284 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:11.284 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:11.285 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:11.285 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:11.285 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:11.286 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:11.286 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:11.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:11.386 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:11.387 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:11.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:11.387 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:11.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:11.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:11.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:11.388 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:11.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:11.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:11.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:11.744 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:12.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:12.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:12.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:12.232 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:12.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:12.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:12.233 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:12.233 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:12.234 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:12.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862885 2021-05-12T13:00:12.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862885 '!=' null 2021-05-12T13:00:12.758 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:12.759 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:12.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:12.759 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:12.851 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:12.852 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:12.852 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:12.852 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:12.853 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:12.853 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:12.853 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:12.853 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:12.853 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:12.854 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:12.854 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:12.854 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:12.854 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:12.855 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:12.855 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:12.855 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:12.855 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:12.855 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:12.856 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:12.856 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:12.856 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:12.856 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:12.857 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:12.857 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:12.857 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:12.857 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:12.857 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:12.858 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:12.858 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:12.858 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:12.860 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:12.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:12.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:12.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:12.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:12.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:12.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:12.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:12.964 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:13.314 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:13.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:13.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:13.314 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:13.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:13.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:13.861 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:13.862 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:13.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:13.863 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:14.496 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241887 2021-05-12T13:00:14.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241887 '!=' null 2021-05-12T13:00:14.497 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:14.498 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:14.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:14.498 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:14.589 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:14.589 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:14.590 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:14.590 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:14.590 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:14.591 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:14.591 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:14.592 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:14.592 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:14.593 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:14.593 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:14.593 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:14.594 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:14.594 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:14.595 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:14.595 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:14.595 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:14.596 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:14.596 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:14.597 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:14.597 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:14.598 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:14.598 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:14.598 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:14.599 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:14.599 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:14.600 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:14.600 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:14.601 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:14.601 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:14.602 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:14.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:14.700 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:14.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:14.701 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:14.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:14.702 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:14.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:14.703 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:15.052 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:15.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:15.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:15.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:15.537 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:15.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:15.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:15.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:15.538 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:15.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:15.539 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:15.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:15.539 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:16.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862864 2021-05-12T13:00:16.033 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862864 '!=' null 2021-05-12T13:00:16.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:16.034 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:16.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:16.034 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:16.127 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:16.127 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:16.127 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:16.128 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:16.128 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:16.128 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:16.129 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:16.129 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:16.129 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:16.129 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:16.129 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:16.130 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:16.130 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:16.130 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:16.130 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:16.131 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:16.131 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:16.131 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:16.131 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:16.132 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:16.132 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:16.132 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:16.133 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:16.133 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:16.133 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:16.133 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:16.134 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:16.134 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:16.134 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:16.134 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:16.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:16.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:16.237 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:16.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:16.237 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:16.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:16.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:16.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:16.238 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:16.596 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:16.596 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:16.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:16.597 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:17.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:17.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:17.139 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:17.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:17.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:17.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:17.140 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:17.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:17.141 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:17.734 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241913 2021-05-12T13:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241913 '!=' null 2021-05-12T13:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:17.735 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:17.736 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:17.828 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:17.828 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:17.828 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:17.829 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:17.829 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:17.829 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:17.829 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:17.830 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:17.830 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:17.830 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:17.830 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:17.831 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:17.831 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:17.831 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:17.831 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:17.831 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:17.832 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:17.832 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:17.832 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:17.832 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:17.833 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:17.833 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:17.833 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:17.834 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:17.834 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:17.834 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:17.834 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:17.835 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:17.835 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:17.836 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:17.838 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:17.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:17.940 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:17.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:17.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:17.941 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:17.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:17.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:17.942 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:18.294 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:18.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:18.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:18.295 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:18.792 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:18.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:18.793 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:18.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:18.794 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:18.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:18.795 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:18.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:18.795 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:19.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T13:00:19.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T13:00:19.277 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:19.278 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:19.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:19.278 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:19.372 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:19.372 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:19.372 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:19.374 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:19.374 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:19.374 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:19.374 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:19.375 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:19.375 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:19.375 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:19.376 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:19.376 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:19.376 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:19.376 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:19.377 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:19.377 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:19.377 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:19.377 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:19.377 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:19.379 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:19.380 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:19.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:19.482 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:19.482 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:19.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:19.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:19.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:19.483 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:19.484 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:19.863 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:19.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:19.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:19.864 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:20.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:20.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:20.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:20.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:20.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:20.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:20.378 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:20.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:20.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:20.867 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241904 2021-05-12T13:00:20.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241904 '!=' null 2021-05-12T13:00:20.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:20.868 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:20.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:20.869 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:20.963 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:20.963 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:20.963 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:20.964 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:20.964 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:20.964 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:20.964 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:20.965 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:20.965 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:20.965 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:20.965 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:20.965 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:20.966 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:20.966 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:20.966 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:20.966 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:20.967 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:20.967 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:20.967 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:20.967 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:20.967 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:20.968 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:20.968 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:20.968 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:20.968 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:20.968 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:20.969 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:20.969 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:20.969 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:20.970 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:20.971 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:21.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:21.073 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:21.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:21.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:21.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:21.074 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:21.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:21.075 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:21.434 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:21.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:21.434 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:21.435 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:21.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:21.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:21.920 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:21.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:21.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:21.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:21.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:21.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:21.922 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:22.403 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862859 2021-05-12T13:00:22.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862859 '!=' null 2021-05-12T13:00:22.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:22.404 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:22.404 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:22.405 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:22.500 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:22.501 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:22.501 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:22.501 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:22.501 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:22.502 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:22.502 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:22.502 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:22.502 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:22.502 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:22.503 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:22.503 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:22.503 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:22.503 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:22.504 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:22.504 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:22.504 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:22.504 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:22.504 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:22.505 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:22.505 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:22.505 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:22.505 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:22.506 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:22.506 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:22.506 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:22.506 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:22.506 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:22.507 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:22.507 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:22.513 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:22.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:22.613 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:22.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:22.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:22.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:22.614 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:22.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:22.615 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:23.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:23.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:23.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:23.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:23.477 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:23.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:23.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:23.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:23.478 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:23.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:23.479 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:23.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:23.479 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:23.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241879 2021-05-12T13:00:23.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241879 '!=' null 2021-05-12T13:00:23.965 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:23.966 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:23.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:23.966 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:24.058 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:24.058 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:24.059 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:24.059 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:24.059 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:24.059 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:24.060 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:24.060 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:24.060 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:24.060 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:24.060 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:24.061 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:24.061 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:24.061 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:24.061 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:24.062 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:24.062 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:24.062 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:24.062 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:24.063 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:24.063 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:24.063 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:24.063 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:24.063 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:24.064 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:24.064 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:24.064 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:24.064 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:24.065 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:24.065 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:24.066 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:24.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:24.168 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:24.169 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:24.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:24.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:24.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:24.170 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:24.171 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:24.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:24.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:24.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:24.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:25.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:25.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:25.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:25.013 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:25.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:25.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:25.014 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:25.014 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:25.015 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:25.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862887 2021-05-12T13:00:25.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862887 '!=' null 2021-05-12T13:00:25.490 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:25.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:25.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:25.491 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:25.584 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:25.584 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:25.585 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:25.585 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:25.585 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:25.585 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:25.586 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:25.586 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:25.586 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:25.587 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:25.587 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:25.587 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:25.587 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:25.588 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:25.588 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:25.588 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:25.588 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:25.589 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:25.589 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:25.589 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:25.590 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:25.590 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:25.590 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:25.591 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:25.591 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:25.591 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:25.591 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:25.592 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:25.592 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:25.592 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:25.593 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:25.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:25.694 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:25.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:25.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:25.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:25.695 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:25.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:25.696 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:26.051 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:26.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:26.052 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:26.053 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:26.527 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:26.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:26.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:26.528 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:26.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:26.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:26.529 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:26.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:26.529 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:27.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862887 2021-05-12T13:00:27.006 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862887 '!=' null 2021-05-12T13:00:27.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:27.007 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:27.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:27.007 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:27.100 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:27.100 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:27.101 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:27.101 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:27.101 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:27.101 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:27.102 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:27.102 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:27.102 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:27.102 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:27.103 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:27.103 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:27.103 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:27.103 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:27.104 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:27.104 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:27.104 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:27.104 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:27.105 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:27.105 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:27.105 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:27.105 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:27.105 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:27.106 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:27.106 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:27.106 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:27.106 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:27.107 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:27.107 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:27.107 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:27.109 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:27.210 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:27.211 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:27.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:27.211 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:27.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:27.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:27.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:27.212 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:27.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:27.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:27.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:27.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:28.067 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:28.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:28.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:28.068 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:28.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:28.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:28.069 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:28.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:28.069 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:28.565 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241946 2021-05-12T13:00:28.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241946 '!=' null 2021-05-12T13:00:28.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:28.566 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:28.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:28.567 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:28.660 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:28.661 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:28.661 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:28.661 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:28.661 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:28.662 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:28.662 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:28.662 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:28.663 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:28.663 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:28.663 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:28.663 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:28.664 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:28.664 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:28.664 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:28.664 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:28.665 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:28.665 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:28.665 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:28.665 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:28.666 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:28.666 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:28.666 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:28.666 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:28.667 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:28.667 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:28.667 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:28.667 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:28.668 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:28.668 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:28.669 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:28.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:28.771 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:28.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:28.771 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:28.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:28.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:28.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:28.772 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:29.135 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:29.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:29.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:29.136 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:29.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:29.803 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:29.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:29.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:29.804 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:29.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:29.805 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:29.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:29.805 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:30.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241946 2021-05-12T13:00:30.286 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241946 '!=' null 2021-05-12T13:00:30.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:30.287 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:30.287 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:30.288 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:30.380 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:30.380 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:30.380 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:30.382 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:30.382 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:30.382 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:30.382 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:30.383 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:30.383 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:30.383 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:30.383 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:30.383 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:30.384 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:30.384 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:30.384 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:30.384 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:30.385 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:30.385 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:30.385 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:30.385 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:30.386 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:30.386 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:30.386 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:30.386 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:30.386 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:30.388 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:30.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:30.491 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:30.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:30.492 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:30.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:30.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:30.493 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:30.494 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:30.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:30.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:30.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:30.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:31.337 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:31.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:31.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:31.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:31.338 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:31.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:31.339 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:31.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989267 2021-05-12T13:00:31.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989267 '!=' null 2021-05-12T13:00:31.820 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:31.821 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:31.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:31.821 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:31.913 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:31.913 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:31.913 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:31.914 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:31.914 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:31.914 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:31.915 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:31.915 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:31.915 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:31.915 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:31.916 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:31.916 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:31.916 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:31.916 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:31.916 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:31.917 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:31.917 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:31.917 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:31.917 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:31.917 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:31.918 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:31.918 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:31.918 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:31.918 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:31.919 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:31.919 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:31.919 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:31.919 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:31.919 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:31.920 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:31.921 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:32.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:32.023 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:32.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:32.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:32.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:32.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:32.024 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:32.025 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:32.377 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:32.378 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:32.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:32.379 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:33.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:33.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:33.143 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:33.144 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:33.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:33.145 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:33.636 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368265 2021-05-12T13:00:33.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368265 '!=' null 2021-05-12T13:00:33.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:33.637 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:33.637 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:33.638 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:33.733 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:33.733 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:33.733 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:33.733 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:33.734 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:33.734 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:33.734 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:33.734 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:33.735 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:33.735 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:33.735 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:33.735 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:33.736 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:33.736 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:33.736 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:33.736 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:33.736 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:33.737 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:33.737 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:33.737 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:33.737 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:33.738 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:33.738 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:33.739 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:33.739 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:33.739 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:33.740 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:33.740 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:33.740 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:33.740 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:33.743 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:33.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:33.845 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:33.845 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:33.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:33.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:33.846 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:33.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:33.847 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:34.258 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:34.258 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:34.259 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:34.749 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:34.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:34.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:34.750 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:34.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:34.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:34.751 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:34.751 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:34.752 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:35.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368265 2021-05-12T13:00:35.441 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368265 '!=' null 2021-05-12T13:00:35.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:35.442 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:35.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:35.442 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:35.534 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:35.535 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:35.535 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:35.535 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:35.536 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:35.536 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:35.536 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:35.536 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:35.536 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:35.537 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:35.537 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:35.537 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:35.537 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:35.538 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:35.538 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:35.538 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:35.538 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:35.539 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:35.539 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:35.539 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:35.539 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:35.540 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:35.540 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:35.540 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:35.540 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:35.540 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:35.541 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:35.541 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:35.541 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:35.541 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:35.543 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:35.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:35.645 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:35.645 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:35.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:35.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:35.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:35.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:35.646 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:36.001 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:36.001 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:36.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:36.002 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:36.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:36.486 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:36.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:36.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:36.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:36.487 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:36.488 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:36.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:36.488 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:36.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989238 2021-05-12T13:00:36.962 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989238 '!=' null 2021-05-12T13:00:36.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:36.963 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:36.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:36.963 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:37.056 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:37.057 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:37.058 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:37.059 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:37.059 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:37.059 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:37.060 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:37.060 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:37.060 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:37.060 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:37.061 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:37.061 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:37.061 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:37.061 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:37.061 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:37.062 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:37.062 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:37.062 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:37.062 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:37.063 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:37.063 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:37.063 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:37.063 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:37.065 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:37.166 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:37.167 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:37.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:37.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:37.167 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:37.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:37.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:37.168 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:37.526 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T13:00:37.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T13:00:37.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T13:00:37.527 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T13:00:38.027 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 3 2021-05-12T13:00:38.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T13:00:38.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T13:00:38.028 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T13:00:38.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T13:00:38.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T13:00:38.029 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T13:00:38.029 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T13:00:38.030 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T13:00:38.516 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T13:00:38.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T13:00:38.517 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T13:00:38.518 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:38.518 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: CEPH_ARGS= 2021-05-12T13:00:38.519 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.185878/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T13:00:38.610 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2021-05-12T13:00:38.610 INFO:tasks.workunit.client.0.smithi090.stdout: "local_reservations": { 2021-05-12T13:00:38.611 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:38.611 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:38.611 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:38.611 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [ 2021-05-12T13:00:38.611 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:38.612 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "1.0", 2021-05-12T13:00:38.612 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:38.612 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:38.612 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:38.612 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:38.613 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "3.0", 2021-05-12T13:00:38.613 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 190, 2021-05-12T13:00:38.613 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:38.613 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:38.614 INFO:tasks.workunit.client.0.smithi090.stdout: { 2021-05-12T13:00:38.614 INFO:tasks.workunit.client.0.smithi090.stdout: "item": "7.0", 2021-05-12T13:00:38.614 INFO:tasks.workunit.client.0.smithi090.stdout: "prio": 255, 2021-05-12T13:00:38.614 INFO:tasks.workunit.client.0.smithi090.stdout: "can_preempt": true 2021-05-12T13:00:38.614 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:38.615 INFO:tasks.workunit.client.0.smithi090.stdout: ] 2021-05-12T13:00:38.615 INFO:tasks.workunit.client.0.smithi090.stdout: }, 2021-05-12T13:00:38.615 INFO:tasks.workunit.client.0.smithi090.stdout: "remote_reservations": { 2021-05-12T13:00:38.615 INFO:tasks.workunit.client.0.smithi090.stdout: "max_allowed": 1000, 2021-05-12T13:00:38.616 INFO:tasks.workunit.client.0.smithi090.stdout: "min_priority": 0, 2021-05-12T13:00:38.616 INFO:tasks.workunit.client.0.smithi090.stdout: "queues": [], 2021-05-12T13:00:38.616 INFO:tasks.workunit.client.0.smithi090.stdout: "in_progress": [] 2021-05-12T13:00:38.616 INFO:tasks.workunit.client.0.smithi090.stdout: } 2021-05-12T13:00:38.616 INFO:tasks.workunit.client.0.smithi090.stdout:} 2021-05-12T13:00:38.618 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T13:00:38.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T13:00:38.720 INFO:tasks.workunit.client.0.smithi090.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T13:00:38.720 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T13:00:38.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T13:00:38.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T13:00:38.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T13:00:38.721 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T13:00:38.722 INFO:tasks.workunit.client.0.smithi090.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T13:00:38.818 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-05-12T13:00:38.820 DEBUG:teuthology.task.console_log:Killing console logger for smithi090 2021-05-12T13:00:38.821 DEBUG:teuthology.exit:Finished running handlers